Sep 17 20:47:13.713276 ip-10-0-131-59 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 20:47:13.713288 ip-10-0-131-59 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 20:47:13.713295 ip-10-0-131-59 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 20:47:13.713451 ip-10-0-131-59 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 20:47:25.113911 ip-10-0-131-59 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 20:47:25.113928 ip-10-0-131-59 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 98ea0079f0514502ad33ad1216e2b44d --
Sep 17 20:49:07.999394 ip-10-0-131-59 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:11.358252 ip-10-0-131-59 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:11.358252 ip-10-0-131-59 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:11.358252 ip-10-0-131-59 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:11.358252 ip-10-0-131-59 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:11.358252 ip-10-0-131-59 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:11.359164 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.358328    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:11.362350 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362335    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.362350 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362349    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362353    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362356    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362361    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362364    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362368    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362371    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362374    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362377    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362379    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362382    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362385    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362387    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362390    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362393    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362397    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362400    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362402    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362405    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362407    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.362409 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362410    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362413    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362415    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362419    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362422    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362425    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362427    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362431    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362433    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362436    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362440    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362444    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362447    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362450    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362453    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362456    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362459    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362462    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362464    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.362939 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362467    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362470    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362472    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362475    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362477    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362480    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362483    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362485    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362488    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362490    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362493    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362495    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362498    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362501    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362503    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362507    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362510    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362513    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362517    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362519    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.363417 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362522    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362525    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362527    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362530    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362533    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362536    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362538    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362541    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362546    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362549    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362559    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362562    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362564    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362567    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362570    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362573    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362575    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362578    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362581    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.363911 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362584    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362588    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362610    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362614    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362616    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362619    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362622    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362625    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362628    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362631    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362634    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362637    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362640    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362643    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362645    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362648    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362651    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362653    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362656    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.362659    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.364379 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363344    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363351    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363353    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363357    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363360    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363363    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363365    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363368    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363371    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363374    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363376    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363379    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363382    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363385    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363388    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363391    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363394    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363397    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363399    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363402    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.364854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363405    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363409    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363412    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363415    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363418    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363421    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363424    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363427    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363430    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363432    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363435    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363438    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363440    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363443    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363446    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363448    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363451    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363454    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363459    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363463    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.365358 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363466    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363469    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363472    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363475    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363478    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363481    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363484    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363487    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363490    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363493    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363496    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363499    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363502    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363505    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363508    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363511    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363513    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363516    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363519    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363522    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.365854 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363524    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363527    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363529    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363532    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363536    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363539    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363541    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363544    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363546    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363552    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363556    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363560    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363564    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363568    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363571    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363574    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363577    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363579    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363583    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.366341 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363586    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363588    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363591    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363594    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363596    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363600    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363603    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363606    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363609    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363611    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363614    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363616    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363619    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363622    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363624    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363627    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363629    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363632    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363634    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.363637    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.366997 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364013    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364018    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364021    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364024    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364026    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364029    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364033    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364035    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364038    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364040    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364043    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364045    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364048    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364050    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364053    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364055    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364058    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364062    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364064    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364067    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.367644 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364070    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364072    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364075    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364078    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364080    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364083    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364086    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364088    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364107    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364110    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364113    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364116    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364119    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364122    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364124    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364127    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364130    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364133    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364136    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364138    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.368155 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364141    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364144    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364146    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364149    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364152    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364154    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364157    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364160    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364162    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364165    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364168    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364171    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364174    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364176    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364180    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364184    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364187    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364189    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364192    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.368651 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364195    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364197    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364200    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364204    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364207    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364210    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364213    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364216    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364219    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364222    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364226    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364229    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364233    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364235    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364238    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364240    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364243    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364245    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364248    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.369174 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364251    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364253    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364256    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364259    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364262    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364265    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364267    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364270    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364273    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364275    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364278    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364280    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364283    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364285    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364288    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364290    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364293    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364295    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364298    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364300    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.369625 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.364302    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364375    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364418    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364426    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364430    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364434    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364438    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364442    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364446    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364449    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364452    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364456    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364459    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364462    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364465    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364470    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364473    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364476    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364479    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364484    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364486    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364489    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364492    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364496    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:11.370161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364500    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364503    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364506    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364509    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364512    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364515    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364518    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364521    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364524    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364528    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364531    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364534    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364537    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364540    2582 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364544    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364975    2582 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364979    2582 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364983    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364986    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364989    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364993    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364996    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.364999    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365005    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365008    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:11.370758 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365011    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365014    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365017    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365020    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365023    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365026    2582 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365029    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365032    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365035    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365038    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365041    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365044    2582 flags.go:64] FLAG: --help="false"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365047    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365050    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365053    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365056    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365059    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365063    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365066    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365068    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365072    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365075    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365078    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365081    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:11.371385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365084    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365087    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365089    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365106    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365109    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365112    2582 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365117    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365121    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365124    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365130    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365133    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365136    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365139    2582 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365142    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365146    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365149    2582 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365151    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365156    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365159    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365163    2582 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365166    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365169    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365172    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365175    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365178    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:11.371983 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365181    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365185    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365192    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365195    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365198    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365201    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365204    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365211    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365214    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365217    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365220    2582 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365223    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365226    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00bb04f2074da373c"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365231    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365240    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365243    2582 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365247    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365250    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365253    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365256    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365259    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365263    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365266    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365269    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365272    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:11.372601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365275    2582 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365278    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365281    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365284    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365287    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365290    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365293    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365296    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365299    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365302    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365305    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365308    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365312    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365315    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365318    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365321    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365326    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365329    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365332    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365336    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365341    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365346    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365348    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365351    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365355    2582 flags.go:64] FLAG: --v="2"
Sep 17 20:49:11.373221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365358    2582 flags.go:64] FLAG: --version="false"
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365363    2582 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365366    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.365370    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365491    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365495    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365499    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365504    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365507    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365510    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365513    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365515    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365518    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365521    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365523    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365526    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365529    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365531    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365534    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365537    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.373839 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365540    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365543    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365545    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365548    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365550    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365553    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365556    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365561    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365566    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365569    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365572    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365576    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365578    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365581    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365584    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365587    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365590    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365593    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365595    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.374331 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365598    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365600    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365603    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365606    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365608    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365611    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365614    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365616    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365619    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365621    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365624    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365627    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365629    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365632    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365634    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365637    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365640    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365642    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365645    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365648    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.374810 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365652    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365656    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365659    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365661    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365664    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365667    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365670    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365672    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365675    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365677    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365680    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365683    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365685    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365688    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365690    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365693    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365696    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365698    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365701    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365704    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.375313 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365706    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365709    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365711    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365714    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365717    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365719    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365722    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365730    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365734    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365737    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365740    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365743    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365747    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365751    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365754    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365757    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365760    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365763    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365765    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365768    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.375817 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365770    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365773    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365776    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.365779    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.366305    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374089    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374119    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374139    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374145    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374197    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374202    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374205    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374209    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374212    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.376348 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374215    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374217    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374220    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374223    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374226    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374228    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374231    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374234    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374237    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374240    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374243    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374245    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374248    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374251    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374253    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374255    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374258    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374260    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374263    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374266    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.376697 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374268    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374271    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374274    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374277    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374280    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374283    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374286    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374289    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374293    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374296    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374298    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374301    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374305    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374309    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374313    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374316    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374319    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374322    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374325    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374328    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.377237 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374331    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374334    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374336    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374339    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374342    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374345    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374348    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374351    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374354    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374357    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374361    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374363    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374366    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374369    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374371    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374374    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374378    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374381    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374383    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.377716 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374386    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374389    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374392    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374394    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374397    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374399    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374403    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374405    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374408    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374410    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374413    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374415    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374418    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374420    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374423    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374426    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374429    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374431    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374434    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374437    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.378243 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374439    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374442    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374445    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374447    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374451    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374454    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374457    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374460    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374464    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374468    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374471    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374474    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374477    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374480    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.378754 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374482    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374487    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374623    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374629    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374633    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374636    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374639    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374642    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374644    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374647    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374650    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374653    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374655    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374658    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374662    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:11.379265 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374665    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374667    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374670    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374673    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374675    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374678    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374680    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374683    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374686    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374688    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374691    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374694    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374697    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374700    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374703    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374706    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374708    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374711    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374714    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374717    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:11.379894 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374720    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374722    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374725    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374728    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374730    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374733    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374736    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374738    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374742    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374745    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374749    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374751    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374755    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374758    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374761    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374763    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374766    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374769    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374772    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374774    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:11.380549 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374777    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374779    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374782    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374785    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374788    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374791    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374793    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374796    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374799    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374801    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374804    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374806    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374809    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374812    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374815    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374817    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374820    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374823    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374826    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374829    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:11.381028 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374831    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374835    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374838    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374841    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374844    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374848    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374851    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374853    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374856    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374858    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374861    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374864    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374866    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374869    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374871    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374874    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374877    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374880    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374883    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:11.381677 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374885    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374888    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374890    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374893    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374896    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374898    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:11.374901    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.374906    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.375011    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.376943    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.377991    2582 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.378125    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.378910    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:11.382167 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.380625    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:11.382525 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.381758    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:11.396921 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.396905    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:11.398548 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.398535    2582 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:11.400129 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.400116    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:11.403171 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.403148    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a9c7afae-dbe3-467d-bfd7-c1ac8ecf6160:/dev/nvme0n1p3 b4bbad14-953c-4f5d-8a83-6fd6d592f12c:/dev/nvme0n1p4]
Sep 17 20:49:11.403233 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.403171    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:11.408725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.408605    2582 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:11.408141428 +0000 UTC m=+1.987377269 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114463 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cef979c3e43f5ca80126ef93b3eb6 SystemUUID:ec2cef97-9c3e-43f5-ca80-126ef93b3eb6 BootID:98ea0079-f051-4502-ad33-ad1216e2b44d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:eb:24:0b:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:eb:24:0b:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:70:75:ac:e6:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:11.408725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.408713    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:11.408879 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.408803    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:11.410366 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.410334    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:11.410503 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.410368    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-59.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:11.411770 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.411760    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:11.411821 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.411773    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:11.411821 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.411781    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:11.411821 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.411805    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:11.413064 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.413055    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:11.416696 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416683    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:11.416818 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416809    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:11.416850 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416841    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:11.416876 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416853    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:11.416876 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416870    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:11.416930 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416880    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:11.416930 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.416895    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:11.417322 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.417304    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:11.417399 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.417327    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:11.417399 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.417350    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:11.419827 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.419797    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:11.422030 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.422016    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:11.422089 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.422047    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427582    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427600    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427607    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427634    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427644    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427650    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427665    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427671    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427679    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427685    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427694    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427701    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427722    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.427728    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:11.430773 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.428253    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:11.432337 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.432319    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:11.432638 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.432620    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-59.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:11.434128 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.434115    2582 server.go:1268] "Started kubelet"
Sep 17 20:49:11.434181 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.434146    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:11.434254 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.434206    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:11.434328 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.434280    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:11.435039 ip-10-0-131-59 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:11.435696 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.435523    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:11.436152 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.436090    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:11.438258 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.438239    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:11.442361 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.442344    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:11.442447 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.442374    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:11.442619 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.442581    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.443032 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443014    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:11.443032 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443014    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:11.443193 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443042    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:11.443193 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443181    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:11.443296 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443199    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:11.443296 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443190    2582 factory.go:55] Registering systemd factory
Sep 17 20:49:11.443296 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443220    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:11.443456 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443436    2582 factory.go:153] Registering CRI-O factory
Sep 17 20:49:11.443456 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443453    2582 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:11.443576 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443513    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:11.443576 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443533    2582 factory.go:103] Registering Raw factory
Sep 17 20:49:11.443576 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443552    2582 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:11.443984 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.443973    2582 manager.go:349] Starting recovery of all containers
Sep 17 20:49:11.462862 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.462828    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 20:49:11.463032 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.463010    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptc6j"
Sep 17 20:49:11.463152 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.463136    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptc6j"
Sep 17 20:49:11.463260 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.463234    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 20:49:11.473860 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.473839    2582 manager.go:354] Recovery completed
Sep 17 20:49:11.477990 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.477976    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.479662 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.479646    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.479740 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.479678    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.479740 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.479692    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.480256 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480240    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:11.480256 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480252    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:11.480369 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480262    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.480369 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480270    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:11.480892 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480879    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.480940 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480901    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.480940 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.480910    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.482369 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.482358    2582 policy_none.go:49] "None policy: Start"
Sep 17 20:49:11.482412 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.482374    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:11.482412 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.482384    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:11.484345 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.484334    2582 policy_none.go:47] "Start"
Sep 17 20:49:11.533721 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.533704    2582 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.533758    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.533772    2582 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.534087    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.534125    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.534224    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.534338    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.534351    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.534916    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:11.547923 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.534961    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.576312 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.576283    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:11.577549 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.577535    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:11.577592 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.577583    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:11.577628 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.577612    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:11.577679 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.577655    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:11.580177 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.580154    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:11.634430 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.634350    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.635390 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.635373    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.635468 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.635405    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.635468 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.635416    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.635468 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.635438    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.643891 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.643873    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.643962 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.643896    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-59.ec2.internal\": node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.662938 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.662911    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.678438 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.678413    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"]
Sep 17 20:49:11.678511 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.678479    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.680473 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.680458    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.680539 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.680487    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.680539 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.680503    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.682700 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.682687    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.682857 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.682845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.682906 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.682872    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.683486 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683471    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.683486 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683484    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.683624 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683501    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.683624 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683506    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.683624 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683514    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.683624 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.683530    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.685901 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.685885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.685980 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.685910    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:11.686546 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.686532    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:11.686639 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.686562    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:11.686639 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.686573    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:11.700692 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.700673    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-59.ec2.internal\" not found" node="ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.704497 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.704482    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-59.ec2.internal\" not found" node="ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.744582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.744554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.744582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.744582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.744726 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.744601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3191f2c20200c49353fb9f7218658ab0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-59.ec2.internal\" (UID: \"3191f2c20200c49353fb9f7218658ab0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.763731 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.763711    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.845295 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.845295 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.845426 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3191f2c20200c49353fb9f7218658ab0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-59.ec2.internal\" (UID: \"3191f2c20200c49353fb9f7218658ab0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.845426 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.845426 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47ee29d28813e1a8ea1c7eb5c0f37ec1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal\" (UID: \"47ee29d28813e1a8ea1c7eb5c0f37ec1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.845426 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:11.845359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3191f2c20200c49353fb9f7218658ab0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-59.ec2.internal\" (UID: \"3191f2c20200c49353fb9f7218658ab0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:11.864403 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.864378    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:11.965262 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:11.965196    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:12.004392 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.004362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:12.008291 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.008272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:12.066465 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:12.066273    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:12.166849 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:12.166815    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:12.267348 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:12.267318    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:12.367830 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:12.367798    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-59.ec2.internal\" not found"
Sep 17 20:49:12.377972 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.377947    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:12.378119 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.378085    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:12.427236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.426515    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:12.443230 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.443207    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:12.443230 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.443226    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:12.454202 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.454179    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:12.455810 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.455790    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal"
Sep 17 20:49:12.456386 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.456369    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:12.465407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.465389    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:12.468952 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.468909    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:11 +0000 UTC" deadline="2028-04-26 14:04:18.156609678 +0000 UTC"
Sep 17 20:49:12.468952 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.468946    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14081h15m5.687667636s"
Sep 17 20:49:12.475913 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.475893    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-42cqz"
Sep 17 20:49:12.483721 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.483701    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-42cqz"
Sep 17 20:49:12.541243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.541172    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:12.626270 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:12.626229    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ee29d28813e1a8ea1c7eb5c0f37ec1.slice/crio-46b7069982334c2afbd3876e9ab66064297ad076f50bb8d63419753df740f973 WatchSource:0}: Error finding container 46b7069982334c2afbd3876e9ab66064297ad076f50bb8d63419753df740f973: Status 404 returned error can't find the container with id 46b7069982334c2afbd3876e9ab66064297ad076f50bb8d63419753df740f973
Sep 17 20:49:12.627007 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:12.626982    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3191f2c20200c49353fb9f7218658ab0.slice/crio-4e3e9abe0dac2b6854d223e208b9dcf5fc2495e07e6d6a4c069bcd88d5b9f4ec WatchSource:0}: Error finding container 4e3e9abe0dac2b6854d223e208b9dcf5fc2495e07e6d6a4c069bcd88d5b9f4ec: Status 404 returned error can't find the container with id 4e3e9abe0dac2b6854d223e208b9dcf5fc2495e07e6d6a4c069bcd88d5b9f4ec
Sep 17 20:49:12.631798 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.631783    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:12.756644 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:12.756612    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:13.419242 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.418946    2582 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:13.426845 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.426169    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:13.426845 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.426609    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-fzp7r","kube-system/konnectivity-agent-t879j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal","openshift-multus/multus-additional-cni-plugins-2852r","openshift-multus/network-metrics-daemon-77s9j","openshift-network-diagnostics/network-check-target-l8chb","openshift-network-operator/iptables-alerter-vg99r","kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dh6z7","openshift-dns/node-resolver-64s7w","openshift-image-registry/node-ca-2p4n6","openshift-multus/multus-g4jlw","openshift-ovn-kubernetes/ovnkube-node-n5kq6"]
Sep 17 20:49:13.429155 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.429129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.431597 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.431374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.431597 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.431374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.431597 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.431378    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c8k8s\""
Sep 17 20:49:13.431597 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.431521    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.434405 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.433935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.434405 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.434062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.434405 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.434198    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:13.434405 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.434302    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.434681 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.434470    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5xqt6\""
Sep 17 20:49:13.434681 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.434524    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:13.436008 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.435990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2snm\""
Sep 17 20:49:13.436131 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.436077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.436200 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.436190    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.436383 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.436362    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:13.436743 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.436726    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.438417 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.438325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.438956 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.438867    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:13.438956 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.438903    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.439634 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.439614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:13.439963 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.438903    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.440576    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.440664    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.440894    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5qh6s\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.440974    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.441203    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.441441 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.441309    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-29hz2\""
Sep 17 20:49:13.441794 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.441776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.442233 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.441945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:13.443977 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.443951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.446318 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.446299    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.446318 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.446302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72xgz\""
Sep 17 20:49:13.446520 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.446492    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.446741 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.446724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:13.446815 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.446796    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:13.446955 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.446919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:13.449297 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.449278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.451844 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.451774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.451949 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.451913    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:13.452376 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.452196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:13.452376 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.452196    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:13.452562 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.452542    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wrntt\""
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-socket-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-conf-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysconfig\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-kubernetes\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-lib-modules\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-netd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbx45\" (UniqueName: \"kubernetes.io/projected/465b7c1a-4120-4138-86bc-fbc4aa842ece-kube-api-access-bbx45\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-multus-certs\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-systemd-units\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.453869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-socket-dir-parent\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-etc-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-run\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-cni-binary-copy\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gth67\" (UniqueName: \"kubernetes.io/projected/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-kube-api-access-gth67\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.453996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6626f74d-2281-432d-a733-40cccd4ccb3f-iptables-alerter-script\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-system-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-multus\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-hostroot\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9l9\" (UniqueName: \"kubernetes.io/projected/573bf01c-5c43-4076-a4bd-daaef1b49f84-kube-api-access-hg9l9\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfh5g\" (UniqueName: \"kubernetes.io/projected/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kube-api-access-sfh5g\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-slash\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-config\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-tuned\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.454587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-device-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-systemd\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-netns\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-ovn\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-env-overrides\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-script-lib\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72js\" (UniqueName: \"kubernetes.io/projected/6626f74d-2281-432d-a733-40cccd4ccb3f-kube-api-access-t72js\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-systemd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-host\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-kubelet\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.454926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcqc\" (UniqueName: \"kubernetes.io/projected/3797aa25-5a62-4ce0-8629-70912fb58b4e-kube-api-access-xqcqc\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-registration-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-sys-fs\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.455629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-cnibin\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-sys\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-kubelet\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-log-socket\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-tmp\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-netns\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-bin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6626f74d-2281-432d-a733-40cccd4ccb3f-host-slash\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-conf\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzqn\" (UniqueName: \"kubernetes.io/projected/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-kube-api-access-4hzqn\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-cnibin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-var-lib-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.456362 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-var-lib-kubelet\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-k8s-cni-cncf-io\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-daemon-config\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-etc-kubernetes\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-node-log\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-bin\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.455989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovn-node-metrics-cert\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.456012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-system-cni-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.457162 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.456030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-modprobe-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.457526 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.457274    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.457526 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.457451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.457526 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.457480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.457692 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.457676    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jxcv\""
Sep 17 20:49:13.459301 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.459274    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:13.459779 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.459762    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lw42v\""
Sep 17 20:49:13.460170 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.460155    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:13.460480 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.460462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:13.486327 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.486254    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:12 +0000 UTC" deadline="2028-06-03 07:20:12.869709898 +0000 UTC"
Sep 17 20:49:13.486327 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.486282    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14986h30m59.383431104s"
Sep 17 20:49:13.543591 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.543561    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:13.557173 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-var-lib-kubelet\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.557257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-k8s-cni-cncf-io\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557193    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-daemon-config\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-etc-kubernetes\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-node-log\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.557257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-bin\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovn-node-metrics-cert\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-k8s-cni-cncf-io\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-system-cni-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-etc-kubernetes\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-bin\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-var-lib-kubelet\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-modprobe-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-system-cni-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-socket-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-node-log\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-hosts-file\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-conf-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bkk\" (UniqueName: \"kubernetes.io/projected/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-kube-api-access-z8bkk\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-conf-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-socket-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-modprobe-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.557556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysconfig\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-kubernetes\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-lib-modules\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysconfig\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-netd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-kubernetes\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbx45\" (UniqueName: \"kubernetes.io/projected/465b7c1a-4120-4138-86bc-fbc4aa842ece-kube-api-access-bbx45\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557609    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-cni-netd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4990bc2f-e0eb-4538-8273-f19cf0ac5339-serviceca\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-lib-modules\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-multus-certs\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfs2\" (UniqueName: \"kubernetes.io/projected/4990bc2f-e0eb-4538-8273-f19cf0ac5339-kube-api-access-qnfs2\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-kubelet-config\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-multus-certs\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.558305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-daemon-config\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-systemd-units\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-systemd-units\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.557990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-socket-dir-parent\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-etc-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-etc-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-run\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558074    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-d\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-socket-dir-parent\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-cni-binary-copy\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-run\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gth67\" (UniqueName: \"kubernetes.io/projected/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-kube-api-access-gth67\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6626f74d-2281-432d-a733-40cccd4ccb3f-iptables-alerter-script\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-system-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559127 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-multus\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-hostroot\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-system-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-tmp-dir\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-multus\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4990bc2f-e0eb-4538-8273-f19cf0ac5339-host\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-hostroot\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9l9\" (UniqueName: \"kubernetes.io/projected/573bf01c-5c43-4076-a4bd-daaef1b49f84-kube-api-access-hg9l9\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.558432    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfh5g\" (UniqueName: \"kubernetes.io/projected/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kube-api-access-sfh5g\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.558492    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:14.058477945 +0000 UTC m=+4.637713779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3797aa25-5a62-4ce0-8629-70912fb58b4e-cni-binary-copy\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2f19303-dfd7-4544-8a8f-ca46440f2278-konnectivity-ca\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.559918 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-slash\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-config\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558656    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-slash\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-tuned\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-device-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6626f74d-2281-432d-a733-40cccd4ccb3f-iptables-alerter-script\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-systemd\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-netns\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-ovn\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-device-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-env-overrides\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-script-lib\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558830    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-ovn\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72js\" (UniqueName: \"kubernetes.io/projected/6626f74d-2281-432d-a733-40cccd4ccb3f-kube-api-access-t72js\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.560568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-systemd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-multus-cni-dir\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-host\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-netns\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-systemd\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-host\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.558969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-systemd\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-kubelet\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcqc\" (UniqueName: \"kubernetes.io/projected/3797aa25-5a62-4ce0-8629-70912fb58b4e-kube-api-access-xqcqc\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-registration-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-sys-fs\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-cnibin\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-kubelet\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559152    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-registration-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.561028 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-sys\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2f19303-dfd7-4544-8a8f-ca46440f2278-agent-certs\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-sys-fs\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-dbus\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-kubelet\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-log-socket\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-tmp\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-etc-selinux\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559301    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-netns\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-sys\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-bin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-run-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-run-ovn-kubernetes\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.561560 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6626f74d-2281-432d-a733-40cccd4ccb3f-host-slash\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-log-socket\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-conf\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/465b7c1a-4120-4138-86bc-fbc4aa842ece-cnibin\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzqn\" (UniqueName: \"kubernetes.io/projected/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-kube-api-access-4hzqn\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-cnibin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-var-lib-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-var-lib-openvswitch\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-cnibin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/465b7c1a-4120-4138-86bc-fbc4aa842ece-cni-binary-copy\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6626f74d-2281-432d-a733-40cccd4ccb3f-host-slash\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-run-netns\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3797aa25-5a62-4ce0-8629-70912fb58b4e-host-var-lib-cni-bin\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/573bf01c-5c43-4076-a4bd-daaef1b49f84-host-kubelet\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-sysctl-conf\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.559774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-config\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.560118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-env-overrides\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.560184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovnkube-script-lib\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.562149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.561493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-tmp\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.562641 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.561547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-etc-tuned\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.562641 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.561718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/573bf01c-5c43-4076-a4bd-daaef1b49f84-ovn-node-metrics-cert\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.572190 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.572169    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:13.572333 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.572195    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:13.572333 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.572209    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:13.573112 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.572403    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:14.072373565 +0000 UTC m=+4.651609400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:13.573112 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.572855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72js\" (UniqueName: \"kubernetes.io/projected/6626f74d-2281-432d-a733-40cccd4ccb3f-kube-api-access-t72js\") pod \"iptables-alerter-vg99r\" (UID: \"6626f74d-2281-432d-a733-40cccd4ccb3f\") " pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.573287 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.573129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gth67\" (UniqueName: \"kubernetes.io/projected/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-kube-api-access-gth67\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:13.573595 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.573577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9l9\" (UniqueName: \"kubernetes.io/projected/573bf01c-5c43-4076-a4bd-daaef1b49f84-kube-api-access-hg9l9\") pod \"ovnkube-node-n5kq6\" (UID: \"573bf01c-5c43-4076-a4bd-daaef1b49f84\") " pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.574188 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.574159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzqn\" (UniqueName: \"kubernetes.io/projected/1d4a40d6-f57f-45df-95a4-0c2a71f58ec6-kube-api-access-4hzqn\") pod \"tuned-dh6z7\" (UID: \"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6\") " pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.574820 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.574790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfh5g\" (UniqueName: \"kubernetes.io/projected/f233c81b-bf65-4ed7-a522-93fdedb01a2b-kube-api-access-sfh5g\") pod \"aws-ebs-csi-driver-node-w55qx\" (UID: \"f233c81b-bf65-4ed7-a522-93fdedb01a2b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.575013 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.574993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcqc\" (UniqueName: \"kubernetes.io/projected/3797aa25-5a62-4ce0-8629-70912fb58b4e-kube-api-access-xqcqc\") pod \"multus-g4jlw\" (UID: \"3797aa25-5a62-4ce0-8629-70912fb58b4e\") " pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.582939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.582901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal" event={"ID":"47ee29d28813e1a8ea1c7eb5c0f37ec1","Type":"ContainerStarted","Data":"46b7069982334c2afbd3876e9ab66064297ad076f50bb8d63419753df740f973"}
Sep 17 20:49:13.584066 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.584034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal" event={"ID":"3191f2c20200c49353fb9f7218658ab0","Type":"ContainerStarted","Data":"4e3e9abe0dac2b6854d223e208b9dcf5fc2495e07e6d6a4c069bcd88d5b9f4ec"}
Sep 17 20:49:13.588670 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.588648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbx45\" (UniqueName: \"kubernetes.io/projected/465b7c1a-4120-4138-86bc-fbc4aa842ece-kube-api-access-bbx45\") pod \"multus-additional-cni-plugins-2852r\" (UID: \"465b7c1a-4120-4138-86bc-fbc4aa842ece\") " pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.660631 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-tmp-dir\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.660631 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4990bc2f-e0eb-4538-8273-f19cf0ac5339-host\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2f19303-dfd7-4544-8a8f-ca46440f2278-konnectivity-ca\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2f19303-dfd7-4544-8a8f-ca46440f2278-agent-certs\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660726    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-dbus\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4990bc2f-e0eb-4538-8273-f19cf0ac5339-host\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-hosts-file\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bkk\" (UniqueName: \"kubernetes.io/projected/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-kube-api-access-z8bkk\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4990bc2f-e0eb-4538-8273-f19cf0ac5339-serviceca\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfs2\" (UniqueName: \"kubernetes.io/projected/4990bc2f-e0eb-4538-8273-f19cf0ac5339-kube-api-access-qnfs2\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.660924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-kubelet-config\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660940    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-tmp-dir\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-kubelet-config\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.660992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3941b7de-2022-492f-a2fa-869389c08f96-dbus\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.661071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-hosts-file\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.661108    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:13.661204    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:14.161184655 +0000 UTC m=+4.740420501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:13.661407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.661325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2f19303-dfd7-4544-8a8f-ca46440f2278-konnectivity-ca\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.661706 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.661599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4990bc2f-e0eb-4538-8273-f19cf0ac5339-serviceca\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.663525 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.663495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2f19303-dfd7-4544-8a8f-ca46440f2278-agent-certs\") pod \"konnectivity-agent-t879j\" (UID: \"a2f19303-dfd7-4544-8a8f-ca46440f2278\") " pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.671482 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.671462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bkk\" (UniqueName: \"kubernetes.io/projected/3549c13d-d5d4-4694-bfe1-f4b0d1d069af-kube-api-access-z8bkk\") pod \"node-resolver-64s7w\" (UID: \"3549c13d-d5d4-4694-bfe1-f4b0d1d069af\") " pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.671601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.671480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfs2\" (UniqueName: \"kubernetes.io/projected/4990bc2f-e0eb-4538-8273-f19cf0ac5339-kube-api-access-qnfs2\") pod \"node-ca-2p4n6\" (UID: \"4990bc2f-e0eb-4538-8273-f19cf0ac5339\") " pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:13.741516 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.741477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dh6z7"
Sep 17 20:49:13.749462 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.749432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4jlw"
Sep 17 20:49:13.760249 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.760220    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx"
Sep 17 20:49:13.767016 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.766992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:13.773829 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.773810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:13.781484 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.781460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2852r"
Sep 17 20:49:13.789156 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.789134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vg99r"
Sep 17 20:49:13.792289 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.792270    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:13.796160 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.796142    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-64s7w"
Sep 17 20:49:13.802747 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:13.802728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2p4n6"
Sep 17 20:49:14.064025 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.063995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:14.064231 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.064116    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:14.064231 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.064184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:15.064167895 +0000 UTC m=+5.643403728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:14.165048 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.164999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:14.165048 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.165052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:14.165293 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165177    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:14.165293 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165208    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:14.165293 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165235    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:15.165221641 +0000 UTC m=+5.744457469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:14.165293 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165242    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:14.165293 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165256    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:14.165529 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.165308    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:15.1652925 +0000 UTC m=+5.744528346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:14.487251 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.487144    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:12 +0000 UTC" deadline="2028-04-08 01:57:24.273575719 +0000 UTC"
Sep 17 20:49:14.487251 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.487208    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13637h8m9.786380831s"
Sep 17 20:49:14.578112 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:14.578075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:14.578249 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:14.578194    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:14.639768 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:14.639724    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6626f74d_2281_432d_a733_40cccd4ccb3f.slice/crio-31e59123deac142590819ce1b142bc3eca8911437fde53da6a648bd46c0e390a WatchSource:0}: Error finding container 31e59123deac142590819ce1b142bc3eca8911437fde53da6a648bd46c0e390a: Status 404 returned error can't find the container with id 31e59123deac142590819ce1b142bc3eca8911437fde53da6a648bd46c0e390a
Sep 17 20:49:14.647318 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:14.643834    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3549c13d_d5d4_4694_bfe1_f4b0d1d069af.slice/crio-041eca5fab5ceb67645b2109464254427a5e3ab492866d49004305f79e1d995e WatchSource:0}: Error finding container 041eca5fab5ceb67645b2109464254427a5e3ab492866d49004305f79e1d995e: Status 404 returned error can't find the container with id 041eca5fab5ceb67645b2109464254427a5e3ab492866d49004305f79e1d995e
Sep 17 20:49:14.649056 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:14.648737    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f19303_dfd7_4544_8a8f_ca46440f2278.slice/crio-6132316826dc778edd73f9311cbbc2c9e449103273bc6c9d1bcd5bfbbc05fc69 WatchSource:0}: Error finding container 6132316826dc778edd73f9311cbbc2c9e449103273bc6c9d1bcd5bfbbc05fc69: Status 404 returned error can't find the container with id 6132316826dc778edd73f9311cbbc2c9e449103273bc6c9d1bcd5bfbbc05fc69
Sep 17 20:49:14.649559 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:14.649454    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4a40d6_f57f_45df_95a4_0c2a71f58ec6.slice/crio-f9c2f0f35fc8abb3d14630a148d2a375e80be115b5560460727031693720c6ce WatchSource:0}: Error finding container f9c2f0f35fc8abb3d14630a148d2a375e80be115b5560460727031693720c6ce: Status 404 returned error can't find the container with id f9c2f0f35fc8abb3d14630a148d2a375e80be115b5560460727031693720c6ce
Sep 17 20:49:15.072827 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.072565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:15.073015 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.072754    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:15.073015 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.072990    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.0729693 +0000 UTC m=+7.652205136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:15.173363 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.173244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:15.173363 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.173303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:15.173600 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173440    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:15.173600 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173503    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.173485092 +0000 UTC m=+7.752720930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:15.173600 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173593    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:15.173758 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173606    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:15.173758 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173619    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:15.173758 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.173657    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:17.173644971 +0000 UTC m=+7.752880802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:15.580770 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.580737    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:15.581497 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.580907    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:15.581497 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.581317    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:15.581497 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:15.581412    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:15.602655 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.602616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerStarted","Data":"c6e4fe7f41320c25b9414ec8f653cd5fc4e1733d9c8bb7473b824a6698c02d2a"}
Sep 17 20:49:15.617973 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.617936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t879j" event={"ID":"a2f19303-dfd7-4544-8a8f-ca46440f2278","Type":"ContainerStarted","Data":"6132316826dc778edd73f9311cbbc2c9e449103273bc6c9d1bcd5bfbbc05fc69"}
Sep 17 20:49:15.630701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.630664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal" event={"ID":"3191f2c20200c49353fb9f7218658ab0","Type":"ContainerStarted","Data":"4144fa48799b69c48117f3efea193bce12e4b6e585d7b116d093a6ea421b14f9"}
Sep 17 20:49:15.639790 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.639680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"0da4b7d7f62432bb6121205d1579a77172d278e61e424c1914921f187abd028c"}
Sep 17 20:49:15.645353 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.645301    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-59.ec2.internal" podStartSLOduration=3.645283255 podStartE2EDuration="3.645283255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:15.64451069 +0000 UTC m=+6.223746541" watchObservedRunningTime="2026-09-17 20:49:15.645283255 +0000 UTC m=+6.224519109"
Sep 17 20:49:15.646117 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.646076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4jlw" event={"ID":"3797aa25-5a62-4ce0-8629-70912fb58b4e","Type":"ContainerStarted","Data":"ea95c710ead690f81803eed4eb7e98f9290d4ee51597853352eb4fe609e4599f"}
Sep 17 20:49:15.652629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.652603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dh6z7" event={"ID":"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6","Type":"ContainerStarted","Data":"f9c2f0f35fc8abb3d14630a148d2a375e80be115b5560460727031693720c6ce"}
Sep 17 20:49:15.654773 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.654739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2p4n6" event={"ID":"4990bc2f-e0eb-4538-8273-f19cf0ac5339","Type":"ContainerStarted","Data":"fe524cc493d1a28c3f3d28db01271b41651daecac6bbbc0be39a2173154bda01"}
Sep 17 20:49:15.658041 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.658017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64s7w" event={"ID":"3549c13d-d5d4-4694-bfe1-f4b0d1d069af","Type":"ContainerStarted","Data":"041eca5fab5ceb67645b2109464254427a5e3ab492866d49004305f79e1d995e"}
Sep 17 20:49:15.659952 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.659925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vg99r" event={"ID":"6626f74d-2281-432d-a733-40cccd4ccb3f","Type":"ContainerStarted","Data":"31e59123deac142590819ce1b142bc3eca8911437fde53da6a648bd46c0e390a"}
Sep 17 20:49:15.665815 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.665787    2582 generic.go:356] "Generic (PLEG): container finished" podID="47ee29d28813e1a8ea1c7eb5c0f37ec1" containerID="65b923552cf9147291bb1f021050d117d187873ee2bf29e862261639ab49515f" exitCode=0
Sep 17 20:49:15.665930 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.665868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal" event={"ID":"47ee29d28813e1a8ea1c7eb5c0f37ec1","Type":"ContainerDied","Data":"65b923552cf9147291bb1f021050d117d187873ee2bf29e862261639ab49515f"}
Sep 17 20:49:15.677169 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:15.677137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx" event={"ID":"f233c81b-bf65-4ed7-a522-93fdedb01a2b","Type":"ContainerStarted","Data":"b0571705fefe9165c8feb73380ea034558d15becb029e77869358e5a8bf08ae0"}
Sep 17 20:49:16.578762 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:16.578685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:16.578965 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:16.578836    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:16.694012 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:16.693974    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal" event={"ID":"47ee29d28813e1a8ea1c7eb5c0f37ec1","Type":"ContainerStarted","Data":"f7f7ba9946e7232ba83ba70537b284ef5cb4fa7a62db2df02dd2e0af8a6f01c9"}
Sep 17 20:49:17.092466 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:17.091805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:17.092466 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.092011    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:17.092466 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.092080    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.092059712 +0000 UTC m=+11.671295566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:17.192254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:17.192307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.192476    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.192541    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.192521037 +0000 UTC m=+11.771756869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.192942    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.192962    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.192975    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:17.193063 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.193020    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:21.193005373 +0000 UTC m=+11.772241207 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:17.578692 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:17.578204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:17.578692 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.578329    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:17.579201 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:17.579028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:17.579201 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:17.579158    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:18.577796 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:18.577764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:18.578243 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:18.577901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:19.578210 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:19.578177    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:19.578670 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:19.578317    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:19.578891 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:19.578842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:19.578994 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:19.578967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:20.578217 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:20.578176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:20.578720 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:20.578312    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:21.126041 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:21.126000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:21.126254 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.126199    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:21.126319 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.126269    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.126250284 +0000 UTC m=+19.705486136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:21.226990 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:21.226953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:21.227218 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:21.227005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:21.227218 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.227165    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:21.227218 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.227201    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:21.227399 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.227231    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:21.227399 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.227244    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:21.227399 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.227233    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.227214127 +0000 UTC m=+19.806449962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:21.236572 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.236534    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:29.236500783 +0000 UTC m=+19.815736611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:21.580375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:21.579922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:21.580375 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.580052    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:21.580375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:21.580122    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:21.580375 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:21.580242    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:22.577800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:22.577768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:22.577975 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:22.577900    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:23.578688 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:23.578359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:23.578688 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:23.578493    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:23.579274 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:23.578892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:23.579274 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:23.579021    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:24.578717 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:24.578688    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:24.579188 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:24.578799    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:25.578913 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:25.578877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:25.579331 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:25.578879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:25.579331 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:25.579021    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:25.579331 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:25.579045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:26.578188 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:26.578154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:26.578375 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:26.578281    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:27.578351 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:27.578318    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:27.578820 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:27.578331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:27.578820 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:27.578442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:27.578820 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:27.578582    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:28.577793 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:28.577765    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:28.577990 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:28.577889    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:29.190901 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:29.190869    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:29.191376 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.191032    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:29.191376 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.191126    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:45.191084265 +0000 UTC m=+35.770320109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:29.291646 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:29.291615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:29.291646 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:29.291650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:29.291900 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291799    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:29.291900 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291822    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:29.291900 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291844    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:29.291900 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291862    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:45.291846658 +0000 UTC m=+35.871082487 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:29.291900 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291874    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:29.292089 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.291927    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:45.291910176 +0000 UTC m=+35.871146010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:29.578126 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:29.578073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:29.578126 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:29.578116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:29.578350 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.578222    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:29.578402 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:29.578355    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:30.578602 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:30.578550    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:30.579014 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:30.578681    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:31.578451 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:31.578420    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:31.578651 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:31.578520    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:31.578651 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:31.578606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:31.579012 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:31.578700    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:32.578870 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.578710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:32.579493 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:32.578947    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:32.729660 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.729621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t879j" event={"ID":"a2f19303-dfd7-4544-8a8f-ca46440f2278","Type":"ContainerStarted","Data":"308268c36534f8cba68455d91141b2d692cb50b9cda17587e7cd0a16e382697b"}
Sep 17 20:49:32.733380 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:49:32.733750 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733712    2582 generic.go:356] "Generic (PLEG): container finished" podID="573bf01c-5c43-4076-a4bd-daaef1b49f84" containerID="964b20459ff26ed5d254e7991f6ace1edc89448d6594f143460116b970b0f035" exitCode=1
Sep 17 20:49:32.733865 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"6be5d9f719340da39a356a67087cc64966a26af0c506fa7c43707c4da3329200"}
Sep 17 20:49:32.733865 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"03c67470ffee8679f70b229eeca5feeabd8e5f939ae6ddcc7e3c059745e45d7f"}
Sep 17 20:49:32.733865 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"b520618a121c24fcb2a4aa0edf064d0b35f316cd6061b65eda35a0cd85ce921d"}
Sep 17 20:49:32.734020 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"5238f4fbead083686ae4aff518029aca783890898c229293d5d78bf746818ca3"}
Sep 17 20:49:32.734020 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerDied","Data":"964b20459ff26ed5d254e7991f6ace1edc89448d6594f143460116b970b0f035"}
Sep 17 20:49:32.734020 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.733971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"706c583a9fcb5aea59f275e5208604dd7f78bd29e15630f550b38d14d03217bb"}
Sep 17 20:49:32.735857 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.735831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4jlw" event={"ID":"3797aa25-5a62-4ce0-8629-70912fb58b4e","Type":"ContainerStarted","Data":"c7b1a9d980ddb9a519787784f162cda1a0847be7979069e32d47551dd04f4320"}
Sep 17 20:49:32.737658 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.737638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dh6z7" event={"ID":"1d4a40d6-f57f-45df-95a4-0c2a71f58ec6","Type":"ContainerStarted","Data":"1df6689f23f85ceacdc5b42ea0d952fb8309b052ac3f9b03c36b0d551f5254ff"}
Sep 17 20:49:32.741794 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.741772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2p4n6" event={"ID":"4990bc2f-e0eb-4538-8273-f19cf0ac5339","Type":"ContainerStarted","Data":"fcf334c588db6c445ce6d7b875b57ac6722cc581f404214f68c0a2c8a484b6c3"}
Sep 17 20:49:32.743337 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.743316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-64s7w" event={"ID":"3549c13d-d5d4-4694-bfe1-f4b0d1d069af","Type":"ContainerStarted","Data":"73ce7f0d7388f90552b87dc795f1c2640a50cd7ad5fb16eac31cced411d03b57"}
Sep 17 20:49:32.744721 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.744703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx" event={"ID":"f233c81b-bf65-4ed7-a522-93fdedb01a2b","Type":"ContainerStarted","Data":"ec1faf17663ac095c5d481e112b52f051966a4f2076656711dc4a93415602506"}
Sep 17 20:49:32.746230 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.746041    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="ee3f98969f48a26e6293460480e8e4f1764c818c446cc43e2cf3443fd41d1ee5" exitCode=0
Sep 17 20:49:32.746230 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.746115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"ee3f98969f48a26e6293460480e8e4f1764c818c446cc43e2cf3443fd41d1ee5"}
Sep 17 20:49:32.755976 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.755930    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t879j" podStartSLOduration=8.684051811 podStartE2EDuration="21.755891475s" totalImagesPullingTime="13.071839664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:32.75586191 +0000 UTC m=+23.335097759" watchObservedRunningTime="2026-09-17 20:49:32.755891475 +0000 UTC m=+23.335127321"
Sep 17 20:49:32.756178 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.756156    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-59.ec2.internal" podStartSLOduration=20.756146651 podStartE2EDuration="20.756146651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:16.707417165 +0000 UTC m=+7.286653016" watchObservedRunningTime="2026-09-17 20:49:32.756146651 +0000 UTC m=+23.335382500"
Sep 17 20:49:32.811795 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.811717    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dh6z7" podStartSLOduration=4.582517889 podStartE2EDuration="21.811697621s" totalImagesPullingTime="17.229179732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:32.811560073 +0000 UTC m=+23.390795925" watchObservedRunningTime="2026-09-17 20:49:32.811697621 +0000 UTC m=+23.390933474"
Sep 17 20:49:32.811940 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.811919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2p4n6" podStartSLOduration=8.738364885 podStartE2EDuration="21.811910321s" totalImagesPullingTime="13.073545436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:32.791318327 +0000 UTC m=+23.370554179" watchObservedRunningTime="2026-09-17 20:49:32.811910321 +0000 UTC m=+23.391146173"
Sep 17 20:49:32.831416 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.831364    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g4jlw" podStartSLOduration=4.562243476 podStartE2EDuration="21.83134837s" totalImagesPullingTime="17.269104894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:32.829920302 +0000 UTC m=+23.409156151" watchObservedRunningTime="2026-09-17 20:49:32.83134837 +0000 UTC m=+23.410584218"
Sep 17 20:49:32.846190 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.846147    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-64s7w" podStartSLOduration=4.615940122 podStartE2EDuration="21.846132079s" totalImagesPullingTime="17.230191957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:32.846017779 +0000 UTC m=+23.425253629" watchObservedRunningTime="2026-09-17 20:49:32.846132079 +0000 UTC m=+23.425367927"
Sep 17 20:49:32.936630 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.936602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:32.938090 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:32.937992    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:33.487904 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.487847    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:49:33.543785 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.543652    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:49:33.487864757Z","UUID":"f0d7ce60-301e-425b-b8a2-1027fe339023","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:49:33.545558 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.545538    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:49:33.545685 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.545572    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:49:33.582181 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.582151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:33.582598 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.582162    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:33.582598 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:33.582271    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:33.582598 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:33.582350    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:33.751020 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.750975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vg99r" event={"ID":"6626f74d-2281-432d-a733-40cccd4ccb3f","Type":"ContainerStarted","Data":"990ad0e935544d7cc924aaa658040eb9d1c936ededfd8b573bf5a6f941807c3a"}
Sep 17 20:49:33.755516 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.755461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx" event={"ID":"f233c81b-bf65-4ed7-a522-93fdedb01a2b","Type":"ContainerStarted","Data":"6ddc4bf4185b5f1ef95cf39262c7dfa505f3a40cb6f396632b4ee17a24b851ee"}
Sep 17 20:49:33.756273 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.756250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:33.756706 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.756688    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t879j"
Sep 17 20:49:33.768003 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:33.767950    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vg99r" podStartSLOduration=5.529504815 podStartE2EDuration="22.767932408s" totalImagesPullingTime="17.238427593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:33.767747236 +0000 UTC m=+24.346983100" watchObservedRunningTime="2026-09-17 20:49:33.767932408 +0000 UTC m=+24.347168257"
Sep 17 20:49:34.578018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:34.577987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:34.578218 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:34.578083    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:34.760764 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:34.760719    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx" event={"ID":"f233c81b-bf65-4ed7-a522-93fdedb01a2b","Type":"ContainerStarted","Data":"baf7a3fd45c1191dfebc1b7f186f40800b13df5583c0b81f1f559d0aa421dad6"}
Sep 17 20:49:34.785721 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:34.785670    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w55qx" podStartSLOduration=4.433748484 podStartE2EDuration="23.785654418s" totalImagesPullingTime="19.351905934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:34.784148327 +0000 UTC m=+25.363384179" watchObservedRunningTime="2026-09-17 20:49:34.785654418 +0000 UTC m=+25.364890268"
Sep 17 20:49:35.578825 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:35.578785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:35.579008 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:35.578843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:35.579008 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:35.578941    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:35.579144 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:35.579073    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:35.766895 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:35.766868    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:49:35.767392 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:35.767338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"0f6223b95b82a73593b796a18e08bb7920f993d25d8f6135d7e16c89fd3bc0f6"}
Sep 17 20:49:36.578303 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:36.578264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:36.578468 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:36.578397    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:37.580916 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.580752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:37.581599 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.580755    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:37.581599 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:37.580982    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:37.581599 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:37.581082    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:37.774578 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.774552    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:49:37.774954 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.774925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"88a3a9f46058ae1ea555540e7e4a528352c899b8cad4e262779f3d8b04604cde"}
Sep 17 20:49:37.775184 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.775165    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:37.775275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.775237    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:37.775382 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.775366    2582 scope.go:117] "RemoveContainer" containerID="964b20459ff26ed5d254e7991f6ace1edc89448d6594f143460116b970b0f035"
Sep 17 20:49:37.776908 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.776883    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="564c8bf0828b9bff3d4bfdd89bdf02f59baf9766c57ec73fd8cf427aea13a1d1" exitCode=0
Sep 17 20:49:37.777008 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.776927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"564c8bf0828b9bff3d4bfdd89bdf02f59baf9766c57ec73fd8cf427aea13a1d1"}
Sep 17 20:49:37.791019 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:37.790989    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:38.578085 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.578053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:38.578261 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:38.578201    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:38.784674 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.784646    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:49:38.785185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.785155    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" event={"ID":"573bf01c-5c43-4076-a4bd-daaef1b49f84","Type":"ContainerStarted","Data":"e8fdc0bc5faf7de4e7f3462aabab74f8f8e34711a737ee9ba72ba9f31f735ee6"}
Sep 17 20:49:38.785706 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.785666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:38.801784 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.801757    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:49:38.820198 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:38.820151    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6" podStartSLOduration=10.579133165 podStartE2EDuration="27.820135554s" totalImagesPullingTime="17.241002389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:38.819727689 +0000 UTC m=+29.398963555" watchObservedRunningTime="2026-09-17 20:49:38.820135554 +0000 UTC m=+29.399371403"
Sep 17 20:49:39.126385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.126348    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fzp7r"]
Sep 17 20:49:39.126556 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.126474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:39.126638 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:39.126614    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:39.129406 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.129296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-77s9j"]
Sep 17 20:49:39.129455 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.129417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:39.129553 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:39.129527    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:39.129990 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.129968    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l8chb"]
Sep 17 20:49:39.130077 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.130063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:39.130187 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:39.130166    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:39.793084 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.793045    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="94f532c1b2efe91361f3d7682f2f21bb5fff7fbfe04138de4d9e10c77cbe2f90" exitCode=0
Sep 17 20:49:39.793612 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:39.793135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"94f532c1b2efe91361f3d7682f2f21bb5fff7fbfe04138de4d9e10c77cbe2f90"}
Sep 17 20:49:40.578079 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:40.578046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:40.578241 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:40.578087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:40.578241 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:40.578059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:40.578241 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:40.578182    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:40.578361 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:40.578241    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:40.578361 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:40.578289    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:42.577989 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:42.577950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:42.578401 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:42.578074    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:42.578401 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:42.578090    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:42.578401 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:42.578208    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:42.578401 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:42.578278    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:42.578401 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:42.578352    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:42.802036 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:42.802007    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="41ab8f58f0b45560abe280da275eb4a45eafaa75bc6a33aa5fd18fb1b6cac0a5" exitCode=0
Sep 17 20:49:42.802207 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:42.802067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"41ab8f58f0b45560abe280da275eb4a45eafaa75bc6a33aa5fd18fb1b6cac0a5"}
Sep 17 20:49:44.578864 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:44.578686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:44.579270 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:44.578724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:44.579270 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:44.578955    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l8chb" podUID="402e4d31-0b67-42ff-954d-104997f52e31"
Sep 17 20:49:44.579270 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:44.579040    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fzp7r" podUID="3941b7de-2022-492f-a2fa-869389c08f96"
Sep 17 20:49:44.579270 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:44.578743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:44.579270 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:44.579158    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:49:44.939832 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:44.939764    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-59.ec2.internal" event="NodeReady"
Sep 17 20:49:44.940011 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:44.939910    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:49:45.003800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.003759    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8hlpm"]
Sep 17 20:49:45.019933 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.019885    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bsjht"]
Sep 17 20:49:45.020130 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.020021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.023273 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.023252    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:49:45.023783 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.023617    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:49:45.023783 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.023653    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:49:45.036262 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.036241    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hlpm"]
Sep 17 20:49:45.036385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.036297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bsjht"]
Sep 17 20:49:45.036385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.036363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.038627 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.038607    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:49:45.038981 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.038856    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:49:45.038981 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.038868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:49:45.038981 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.038884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b5xsj\""
Sep 17 20:49:45.102338 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.102304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4874db31-3321-40bf-8eed-2b111959c3ac-config-volume\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.102549 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.102347    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.102549 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.102425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4874db31-3321-40bf-8eed-2b111959c3ac-tmp-dir\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.102549 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.102535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mkj\" (UniqueName: \"kubernetes.io/projected/4874db31-3321-40bf-8eed-2b111959c3ac-kube-api-access-m8mkj\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.203619 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:45.203619 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4874db31-3321-40bf-8eed-2b111959c3ac-config-volume\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4874db31-3321-40bf-8eed-2b111959c3ac-tmp-dir\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/445ed9f2-bb49-4023-ad4c-ef2b81082509-cert\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9lb\" (UniqueName: \"kubernetes.io/projected/445ed9f2-bb49-4023-ad4c-ef2b81082509-kube-api-access-ct9lb\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.203724    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.203735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mkj\" (UniqueName: \"kubernetes.io/projected/4874db31-3321-40bf-8eed-2b111959c3ac-kube-api-access-m8mkj\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.203803    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.203854    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:17.203828192 +0000 UTC m=+67.783064038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:45.203909 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.203904    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:49:45.703884118 +0000 UTC m=+36.283119951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:45.204334 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.204247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4874db31-3321-40bf-8eed-2b111959c3ac-tmp-dir\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.204334 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.204312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4874db31-3321-40bf-8eed-2b111959c3ac-config-volume\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.214512 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.214489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mkj\" (UniqueName: \"kubernetes.io/projected/4874db31-3321-40bf-8eed-2b111959c3ac-kube-api-access-m8mkj\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.304836 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.304802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/445ed9f2-bb49-4023-ad4c-ef2b81082509-cert\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.304836 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.304845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9lb\" (UniqueName: \"kubernetes.io/projected/445ed9f2-bb49-4023-ad4c-ef2b81082509-kube-api-access-ct9lb\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.305113 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.305070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:45.305181 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.305134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:45.305247 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305227    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:45.305289 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305257    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:45.305289 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305270    2582 projected.go:196] Error preparing data for projected volume kube-api-access-df9t4 for pod openshift-network-diagnostics/network-check-target-l8chb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:45.305289 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305280    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:45.305386 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305353    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4 podName:402e4d31-0b67-42ff-954d-104997f52e31 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:17.305332881 +0000 UTC m=+67.884568729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-df9t4" (UniqueName: "kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4") pod "network-check-target-l8chb" (UID: "402e4d31-0b67-42ff-954d-104997f52e31") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:45.305386 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.305380    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret podName:3941b7de-2022-492f-a2fa-869389c08f96 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:17.305361418 +0000 UTC m=+67.884597247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret") pod "global-pull-secret-syncer-fzp7r" (UID: "3941b7de-2022-492f-a2fa-869389c08f96") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:45.307890 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.307865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/445ed9f2-bb49-4023-ad4c-ef2b81082509-cert\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.314628 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.314602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9lb\" (UniqueName: \"kubernetes.io/projected/445ed9f2-bb49-4023-ad4c-ef2b81082509-kube-api-access-ct9lb\") pod \"ingress-canary-bsjht\" (UID: \"445ed9f2-bb49-4023-ad4c-ef2b81082509\") " pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.349102 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.349073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bsjht"
Sep 17 20:49:45.682237 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.682206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bsjht"]
Sep 17 20:49:45.708531 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:45.708472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:45.708649 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.708582    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:45.708649 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:45.708639    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:49:46.708623273 +0000 UTC m=+37.287859120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:46.577858 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.577820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:49:46.577858 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.577842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:49:46.578243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.577842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:49:46.583406 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.583384    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:49:46.583530 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.583390    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:49:46.583927 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.583908    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:49:46.584048 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.583972    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:49:46.584223 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.584196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:49:46.584223 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.584222    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:49:46.717253 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:46.717213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:46.717709 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:46.717359    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:46.717709 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:46.717433    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:49:48.717414713 +0000 UTC m=+39.296650541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:47.227910 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:49:47.227878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445ed9f2_bb49_4023_ad4c_ef2b81082509.slice/crio-8f5ae236e2491b197dbe9e05103ed6dc49fd307d874d56ab06219a0ff3f90909 WatchSource:0}: Error finding container 8f5ae236e2491b197dbe9e05103ed6dc49fd307d874d56ab06219a0ff3f90909: Status 404 returned error can't find the container with id 8f5ae236e2491b197dbe9e05103ed6dc49fd307d874d56ab06219a0ff3f90909
Sep 17 20:49:47.819688 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:47.819395    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="fc64a8e46ad351c63989cf871d9d5265236ccd97dc4975605396b7c911e2f9cc" exitCode=0
Sep 17 20:49:47.820137 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:47.819566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"fc64a8e46ad351c63989cf871d9d5265236ccd97dc4975605396b7c911e2f9cc"}
Sep 17 20:49:47.821316 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:47.821267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bsjht" event={"ID":"445ed9f2-bb49-4023-ad4c-ef2b81082509","Type":"ContainerStarted","Data":"8f5ae236e2491b197dbe9e05103ed6dc49fd307d874d56ab06219a0ff3f90909"}
Sep 17 20:49:48.734438 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:48.734404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:48.734628 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:48.734542    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:48.734628 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:48.734594    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.734580741 +0000 UTC m=+43.313816569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:49:48.827976 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:48.827944    2582 generic.go:356] "Generic (PLEG): container finished" podID="465b7c1a-4120-4138-86bc-fbc4aa842ece" containerID="9c933b4a9d29176a5beeaca55fb75b0865e163a71ed9f0c74ed6fea65ff51607" exitCode=0
Sep 17 20:49:48.828407 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:48.827993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerDied","Data":"9c933b4a9d29176a5beeaca55fb75b0865e163a71ed9f0c74ed6fea65ff51607"}
Sep 17 20:49:49.833502 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:49.833416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2852r" event={"ID":"465b7c1a-4120-4138-86bc-fbc4aa842ece","Type":"ContainerStarted","Data":"bf1a9c335f00ce39537c2b5149bd365ff8ba31de3e58a9b9f69646d1b23461e8"}
Sep 17 20:49:49.834910 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:49.834885    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bsjht" event={"ID":"445ed9f2-bb49-4023-ad4c-ef2b81082509","Type":"ContainerStarted","Data":"8c8e3a513ece1619c5aa006f22fd06d8c2d5d7c069cdbdb644c68707dc933a04"}
Sep 17 20:49:49.855847 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:49.855803    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2852r" podStartSLOduration=8.594675673 podStartE2EDuration="38.855787839s" totalImagesPullingTime="30.261112166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:49.854688741 +0000 UTC m=+40.433924603" watchObservedRunningTime="2026-09-17 20:49:49.855787839 +0000 UTC m=+40.435023689"
Sep 17 20:49:52.765365 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:49:52.765310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:49:52.765755 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:52.765456    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:49:52.765755 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:49:52.765517    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:50:00.765502173 +0000 UTC m=+51.344738001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:00.826986 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:00.826941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:50:00.827434 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:00.827079    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:00.827434 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:00.827174    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:50:16.827158005 +0000 UTC m=+67.406393835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:10.812892 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:10.812861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n5kq6"
Sep 17 20:50:10.845216 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:10.845167    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bsjht" podStartSLOduration=24.559418328 podStartE2EDuration="26.84515094s" totalImagesPullingTime="2.285732612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:49.873828242 +0000 UTC m=+40.453064093" watchObservedRunningTime="2026-09-17 20:50:10.84515094 +0000 UTC m=+61.424386784"
Sep 17 20:50:13.929813 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.929781    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"]
Sep 17 20:50:13.932689 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.932667    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:13.934846 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.934823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 20:50:13.935622 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.935600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 20:50:13.935705 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.935621    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 20:50:13.935705 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.935604    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 20:50:13.942840 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.942814    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"]
Sep 17 20:50:13.943397 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.943375    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"]
Sep 17 20:50:13.945054 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.945038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:13.947487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.947471    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 20:50:13.947806 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.947790    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bstbj\""
Sep 17 20:50:13.957641 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:13.957606    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"]
Sep 17 20:50:14.014422 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.014390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb195a78-89a4-4665-93f7-c6abe5b329d8-tmp\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.014422 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.014428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb195a78-89a4-4665-93f7-c6abe5b329d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.014661 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.014501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x958f\" (UniqueName: \"kubernetes.io/projected/fb195a78-89a4-4665-93f7-c6abe5b329d8-kube-api-access-x958f\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.014697 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.014656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6eb382e8-564d-4304-931f-2a8171e12edc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.014697 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.014682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd66b\" (UniqueName: \"kubernetes.io/projected/6eb382e8-564d-4304-931f-2a8171e12edc-kube-api-access-rd66b\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.115722 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.115691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6eb382e8-564d-4304-931f-2a8171e12edc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.115722 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.115724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd66b\" (UniqueName: \"kubernetes.io/projected/6eb382e8-564d-4304-931f-2a8171e12edc-kube-api-access-rd66b\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.115850 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.115759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb195a78-89a4-4665-93f7-c6abe5b329d8-tmp\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.115850 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.115779    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb195a78-89a4-4665-93f7-c6abe5b329d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.115945 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.115920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x958f\" (UniqueName: \"kubernetes.io/projected/fb195a78-89a4-4665-93f7-c6abe5b329d8-kube-api-access-x958f\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.116208 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.116186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb195a78-89a4-4665-93f7-c6abe5b329d8-tmp\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.119045 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.119017    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6eb382e8-564d-4304-931f-2a8171e12edc-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.119045 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.119034    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fb195a78-89a4-4665-93f7-c6abe5b329d8-klusterlet-config\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.124361 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.124339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x958f\" (UniqueName: \"kubernetes.io/projected/fb195a78-89a4-4665-93f7-c6abe5b329d8-kube-api-access-x958f\") pod \"klusterlet-addon-workmgr-848967c6d-f2cr6\" (UID: \"fb195a78-89a4-4665-93f7-c6abe5b329d8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.124483 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.124447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd66b\" (UniqueName: \"kubernetes.io/projected/6eb382e8-564d-4304-931f-2a8171e12edc-kube-api-access-rd66b\") pod \"managed-serviceaccount-addon-agent-56fd9c68b8-khgr5\" (UID: \"6eb382e8-564d-4304-931f-2a8171e12edc\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.242447 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.242358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:14.263287 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.263262    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"
Sep 17 20:50:14.526889 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.526859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"]
Sep 17 20:50:14.530679 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:50:14.530649    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb195a78_89a4_4665_93f7_c6abe5b329d8.slice/crio-6bc4e3989cba13026c1f21bca7eb618430a8d89f3c42ffe1ddd37e9241334da7 WatchSource:0}: Error finding container 6bc4e3989cba13026c1f21bca7eb618430a8d89f3c42ffe1ddd37e9241334da7: Status 404 returned error can't find the container with id 6bc4e3989cba13026c1f21bca7eb618430a8d89f3c42ffe1ddd37e9241334da7
Sep 17 20:50:14.544924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.544897    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5"]
Sep 17 20:50:14.547478 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:50:14.547457    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eb382e8_564d_4304_931f_2a8171e12edc.slice/crio-ec06f9ea67bdc6b90bea7d2a3f5213a249d6b803159e62ea7d094828b4a8b700 WatchSource:0}: Error finding container ec06f9ea67bdc6b90bea7d2a3f5213a249d6b803159e62ea7d094828b4a8b700: Status 404 returned error can't find the container with id ec06f9ea67bdc6b90bea7d2a3f5213a249d6b803159e62ea7d094828b4a8b700
Sep 17 20:50:14.893799 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.893706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5" event={"ID":"6eb382e8-564d-4304-931f-2a8171e12edc","Type":"ContainerStarted","Data":"ec06f9ea67bdc6b90bea7d2a3f5213a249d6b803159e62ea7d094828b4a8b700"}
Sep 17 20:50:14.894851 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:14.894826    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6" event={"ID":"fb195a78-89a4-4665-93f7-c6abe5b329d8","Type":"ContainerStarted","Data":"6bc4e3989cba13026c1f21bca7eb618430a8d89f3c42ffe1ddd37e9241334da7"}
Sep 17 20:50:16.838258 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:16.838214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:50:16.838889 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:16.838390    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:16.838889 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:16.838462    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:50:48.838441532 +0000 UTC m=+99.417677361 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:17.241581 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.241463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:50:17.244183 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.244148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:17.252251 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:17.252222    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:17.252454 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:17.252295    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.252278124 +0000 UTC m=+131.831513952 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : secret "metrics-daemon-secret" not found
Sep 17 20:50:17.342941 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.342444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:50:17.342941 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.342499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:50:17.345048 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.345023    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:17.345179 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.345027    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:17.355236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.355176    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:17.356021 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.355997    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3941b7de-2022-492f-a2fa-869389c08f96-original-pull-secret\") pod \"global-pull-secret-syncer-fzp7r\" (UID: \"3941b7de-2022-492f-a2fa-869389c08f96\") " pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:50:17.368284 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.368256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9t4\" (UniqueName: \"kubernetes.io/projected/402e4d31-0b67-42ff-954d-104997f52e31-kube-api-access-df9t4\") pod \"network-check-target-l8chb\" (UID: \"402e4d31-0b67-42ff-954d-104997f52e31\") " pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:50:17.490169 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.490135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fzp7r"
Sep 17 20:50:17.499649 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.499626    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vnpsw\""
Sep 17 20:50:17.506781 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:17.506758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:50:19.577035 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.576962    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fzp7r"]
Sep 17 20:50:19.580234 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:50:19.580193    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3941b7de_2022_492f_a2fa_869389c08f96.slice/crio-688f89bb36593b0db7b264b85b4be2a5d84c6907401a29500149077e9f665822 WatchSource:0}: Error finding container 688f89bb36593b0db7b264b85b4be2a5d84c6907401a29500149077e9f665822: Status 404 returned error can't find the container with id 688f89bb36593b0db7b264b85b4be2a5d84c6907401a29500149077e9f665822
Sep 17 20:50:19.582312 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.582294    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l8chb"]
Sep 17 20:50:19.583291 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:50:19.583271    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402e4d31_0b67_42ff_954d_104997f52e31.slice/crio-2d0574a38f5df708858a3cb14e8e83fad2d73015147479cb5cb880f11b989576 WatchSource:0}: Error finding container 2d0574a38f5df708858a3cb14e8e83fad2d73015147479cb5cb880f11b989576: Status 404 returned error can't find the container with id 2d0574a38f5df708858a3cb14e8e83fad2d73015147479cb5cb880f11b989576
Sep 17 20:50:19.912973 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.912882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fzp7r" event={"ID":"3941b7de-2022-492f-a2fa-869389c08f96","Type":"ContainerStarted","Data":"688f89bb36593b0db7b264b85b4be2a5d84c6907401a29500149077e9f665822"}
Sep 17 20:50:19.914121 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.914076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5" event={"ID":"6eb382e8-564d-4304-931f-2a8171e12edc","Type":"ContainerStarted","Data":"7409a06fe83696d294605201c8d1c42fc9b534d43dc8020570959057482317ec"}
Sep 17 20:50:19.915498 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.915474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6" event={"ID":"fb195a78-89a4-4665-93f7-c6abe5b329d8","Type":"ContainerStarted","Data":"b68b048a91e7c6a00bb46ba67b81358b59f954c1e752958997e94a4b69f0e217"}
Sep 17 20:50:19.915623 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.915520    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:19.916601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.916582    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l8chb" event={"ID":"402e4d31-0b67-42ff-954d-104997f52e31","Type":"ContainerStarted","Data":"2d0574a38f5df708858a3cb14e8e83fad2d73015147479cb5cb880f11b989576"}
Sep 17 20:50:19.917245 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.917228    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6"
Sep 17 20:50:19.932153 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.932109    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-56fd9c68b8-khgr5" podStartSLOduration=2.168162431 podStartE2EDuration="6.932081794s" totalImagesPullingTime="4.763919363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:19.931591671 +0000 UTC m=+70.510827523" watchObservedRunningTime="2026-09-17 20:50:19.932081794 +0000 UTC m=+70.511317643"
Sep 17 20:50:19.954595 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:19.954526    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848967c6d-f2cr6" podStartSLOduration=2.159150609 podStartE2EDuration="6.954504699s" totalImagesPullingTime="4.79535409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:19.953321794 +0000 UTC m=+70.532557643" watchObservedRunningTime="2026-09-17 20:50:19.954504699 +0000 UTC m=+70.533740551"
Sep 17 20:50:24.933156 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:24.933116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l8chb" event={"ID":"402e4d31-0b67-42ff-954d-104997f52e31","Type":"ContainerStarted","Data":"c2f36ba4e6f0e23b06ff66061a09358725914ecea2370b00547735ecc29d016c"}
Sep 17 20:50:24.933601 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:24.933224    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:50:24.934628 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:24.934605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fzp7r" event={"ID":"3941b7de-2022-492f-a2fa-869389c08f96","Type":"ContainerStarted","Data":"aeb752b9fb93a70200a3e44e7490a0dd8007822d38a50a09a748d4d9a8324c63"}
Sep 17 20:50:24.948146 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:24.948081    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l8chb" podStartSLOduration=69.628195916 podStartE2EDuration="1m13.948068449s" totalImagesPullingTime="4.319872533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:24.947774529 +0000 UTC m=+75.527010379" watchObservedRunningTime="2026-09-17 20:50:24.948068449 +0000 UTC m=+75.527304319"
Sep 17 20:50:24.961921 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:24.961877    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fzp7r" podStartSLOduration=69.636198412 podStartE2EDuration="1m13.961861492s" totalImagesPullingTime="4.32566308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:24.961735805 +0000 UTC m=+75.540971656" watchObservedRunningTime="2026-09-17 20:50:24.961861492 +0000 UTC m=+75.541097343"
Sep 17 20:50:48.852709 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:48.852671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:50:48.853243 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:48.852858    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:48.853243 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:50:48.852946    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls podName:4874db31-3321-40bf-8eed-2b111959c3ac nodeName:}" failed. No retries permitted until 2026-09-17 20:51:52.852921817 +0000 UTC m=+163.432157660 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls") pod "dns-default-8hlpm" (UID: "4874db31-3321-40bf-8eed-2b111959c3ac") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:55.939895 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:50:55.939866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l8chb"
Sep 17 20:51:09.111991 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.111942    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2cnjp"]
Sep 17 20:51:09.115056 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.115032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.117129 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.117106    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:09.117252 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.117160    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:09.117252 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.117187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:09.117995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.117974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:09.117995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.117980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9nvv\""
Sep 17 20:51:09.121263 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.121198    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cnjp"]
Sep 17 20:51:09.193759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.193721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-data-volume\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.193759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.193759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.193995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.193785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.193995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.193824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccfc\" (UniqueName: \"kubernetes.io/projected/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-api-access-jccfc\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.193995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.193871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-crio-socket\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295056 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-data-volume\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295056 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295082    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccfc\" (UniqueName: \"kubernetes.io/projected/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-api-access-jccfc\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-crio-socket\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.295182    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-crio-socket\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.295294 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.295253    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.295555 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.295265    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.795247771 +0000 UTC m=+120.374483619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.295555 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.295339    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.795322251 +0000 UTC m=+120.374558084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.295555 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.295440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-data-volume\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.305546 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.305524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccfc\" (UniqueName: \"kubernetes.io/projected/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-api-access-jccfc\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.799321 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.799278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.799321 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:09.799320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:09.799556 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.799414    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.799556 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.799469    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.799556 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.799484    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.799463885 +0000 UTC m=+121.378699733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.799556 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:09.799513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:10.799502389 +0000 UTC m=+121.378738233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:10.805541 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:10.805507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:10.805541 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:10.805545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:10.805922 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:10.805636    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.805922 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:10.805670    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:10.805922 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:10.805705    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.805687061 +0000 UTC m=+123.384922905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:10.805922 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:10.805719    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:12.805713201 +0000 UTC m=+123.384949029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:11.039137 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.039112    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64s7w_3549c13d-d5d4-4694-bfe1-f4b0d1d069af/dns-node-resolver/0.log"
Sep 17 20:51:11.367822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.367795    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"]
Sep 17 20:51:11.369908 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.369893    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"
Sep 17 20:51:11.372633 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.372612    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:11.373393 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.373379    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:11.373446 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.373387    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d5kzf\""
Sep 17 20:51:11.381607 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.381582    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"]
Sep 17 20:51:11.510567 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.510537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdvg6\" (UniqueName: \"kubernetes.io/projected/d49208e2-0b60-4986-b5e9-674e612579e6-kube-api-access-gdvg6\") pod \"volume-data-source-validator-5c4db8bdf9-jpflf\" (UID: \"d49208e2-0b60-4986-b5e9-674e612579e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"
Sep 17 20:51:11.611259 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.611214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvg6\" (UniqueName: \"kubernetes.io/projected/d49208e2-0b60-4986-b5e9-674e612579e6-kube-api-access-gdvg6\") pod \"volume-data-source-validator-5c4db8bdf9-jpflf\" (UID: \"d49208e2-0b60-4986-b5e9-674e612579e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"
Sep 17 20:51:11.624307 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.624239    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:11.633756 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.633735    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:11.644647 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.644626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdvg6\" (UniqueName: \"kubernetes.io/projected/d49208e2-0b60-4986-b5e9-674e612579e6-kube-api-access-gdvg6\") pod \"volume-data-source-validator-5c4db8bdf9-jpflf\" (UID: \"d49208e2-0b60-4986-b5e9-674e612579e6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"
Sep 17 20:51:11.681334 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.681307    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d5kzf\""
Sep 17 20:51:11.689136 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.689115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"
Sep 17 20:51:11.839695 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.839668    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2p4n6_4990bc2f-e0eb-4538-8273-f19cf0ac5339/node-ca/0.log"
Sep 17 20:51:11.955938 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:11.955862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf"]
Sep 17 20:51:11.959351 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:11.959322    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49208e2_0b60_4986_b5e9_674e612579e6.slice/crio-cabe499c29e5b3b30ddd7598f1c21ba0012d9ea1bf2db5ed79f0a383170d3301 WatchSource:0}: Error finding container cabe499c29e5b3b30ddd7598f1c21ba0012d9ea1bf2db5ed79f0a383170d3301: Status 404 returned error can't find the container with id cabe499c29e5b3b30ddd7598f1c21ba0012d9ea1bf2db5ed79f0a383170d3301
Sep 17 20:51:12.066429 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.066396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf" event={"ID":"d49208e2-0b60-4986-b5e9-674e612579e6","Type":"ContainerStarted","Data":"cabe499c29e5b3b30ddd7598f1c21ba0012d9ea1bf2db5ed79f0a383170d3301"}
Sep 17 20:51:12.413869 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.413833    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-bdpxp"]
Sep 17 20:51:12.416239 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.416218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.419619 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.419602    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 20:51:12.419726 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.419665    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cpd9s\""
Sep 17 20:51:12.419924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.419908    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 20:51:12.419995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.419942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 20:51:12.419995 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.419977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:12.427626 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.427597    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 20:51:12.429221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.429200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-bdpxp"]
Sep 17 20:51:12.447712 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.447690    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bsjht_445ed9f2-bb49-4023-ad4c-ef2b81082509/serve-healthcheck-canary/0.log"
Sep 17 20:51:12.519639 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.519607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-config\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.519806 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.519678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-serving-cert\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.519806 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.519728    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtth\" (UniqueName: \"kubernetes.io/projected/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-kube-api-access-tmtth\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.519806 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.519753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-trusted-ca\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.620936 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.620901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-serving-cert\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.621157 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.620945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtth\" (UniqueName: \"kubernetes.io/projected/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-kube-api-access-tmtth\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.621157 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.620972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-trusted-ca\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.621157 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.621106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-config\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.622143 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.622118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-config\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.622274 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.622204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-trusted-ca\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.623727 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.623706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-serving-cert\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.631766 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.631747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtth\" (UniqueName: \"kubernetes.io/projected/4f3e9553-733b-4bcf-bb90-e20d37ba7efb-kube-api-access-tmtth\") pod \"console-operator-fd4bb46c-bdpxp\" (UID: \"4f3e9553-733b-4bcf-bb90-e20d37ba7efb\") " pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.729281 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.729198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:12.822656 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.822596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:12.822656 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:12.822654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:12.822866 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:12.822747    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:12.822866 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:12.822796    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.822782994 +0000 UTC m=+127.402018823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:12.822866 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:12.822742    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:12.822999 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:12.822910    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:16.822887398 +0000 UTC m=+127.402123260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:13.009367 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:13.009333    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-bdpxp"]
Sep 17 20:51:13.018002 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:13.017970    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3e9553_733b_4bcf_bb90_e20d37ba7efb.slice/crio-c9f5fad12181e20ca8b3b57b75003df106247ca1f39883069659d9237989bdf8 WatchSource:0}: Error finding container c9f5fad12181e20ca8b3b57b75003df106247ca1f39883069659d9237989bdf8: Status 404 returned error can't find the container with id c9f5fad12181e20ca8b3b57b75003df106247ca1f39883069659d9237989bdf8
Sep 17 20:51:13.070461 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:13.070427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" event={"ID":"4f3e9553-733b-4bcf-bb90-e20d37ba7efb","Type":"ContainerStarted","Data":"c9f5fad12181e20ca8b3b57b75003df106247ca1f39883069659d9237989bdf8"}
Sep 17 20:51:14.075650 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:14.075618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf" event={"ID":"d49208e2-0b60-4986-b5e9-674e612579e6","Type":"ContainerStarted","Data":"17ccec79133f5131a71c6d400f04aa94ce38ceb8337355212e93cdc1df4f424e"}
Sep 17 20:51:14.093710 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:14.093650    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-jpflf" podStartSLOduration=2.011261927 podStartE2EDuration="3.093626564s" totalImagesPullingTime="1.082364637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:14.092363314 +0000 UTC m=+124.671599167" watchObservedRunningTime="2026-09-17 20:51:14.093626564 +0000 UTC m=+124.672862415"
Sep 17 20:51:16.082708 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.082677    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/0.log"
Sep 17 20:51:16.083211 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.082717    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb" containerID="b66f49f7a5c37f6a8a73c77d3e7d9a67ae2271816ea0714fe1f9b8147ea84107" exitCode=255
Sep 17 20:51:16.083211 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.082777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" event={"ID":"4f3e9553-733b-4bcf-bb90-e20d37ba7efb","Type":"ContainerDied","Data":"b66f49f7a5c37f6a8a73c77d3e7d9a67ae2271816ea0714fe1f9b8147ea84107"}
Sep 17 20:51:16.083211 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.083052    2582 scope.go:117] "RemoveContainer" containerID="b66f49f7a5c37f6a8a73c77d3e7d9a67ae2271816ea0714fe1f9b8147ea84107"
Sep 17 20:51:16.852764 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.852732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:16.852764 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:16.852768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:16.853020 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:16.852853    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:16.853020 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:16.852901    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:16.853020 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:16.852952    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.852910286 +0000 UTC m=+135.432146128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:16.853020 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:16.852971    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.852962015 +0000 UTC m=+135.432197844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:17.087444 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.087412    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/1.log"
Sep 17 20:51:17.087819 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.087797    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/0.log"
Sep 17 20:51:17.087873 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.087832    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21" exitCode=255
Sep 17 20:51:17.087873 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.087861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" event={"ID":"4f3e9553-733b-4bcf-bb90-e20d37ba7efb","Type":"ContainerDied","Data":"45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"}
Sep 17 20:51:17.087971 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.087929    2582 scope.go:117] "RemoveContainer" containerID="b66f49f7a5c37f6a8a73c77d3e7d9a67ae2271816ea0714fe1f9b8147ea84107"
Sep 17 20:51:17.088188 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:17.088168    2582 scope.go:117] "RemoveContainer" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"
Sep 17 20:51:17.088406 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:17.088382    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-bdpxp_openshift-console-operator(4f3e9553-733b-4bcf-bb90-e20d37ba7efb)\"" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podUID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb"
Sep 17 20:51:18.092511 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:18.092483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/1.log"
Sep 17 20:51:18.092932 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:18.092786    2582 scope.go:117] "RemoveContainer" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"
Sep 17 20:51:18.092992 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:18.092944    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-bdpxp_openshift-console-operator(4f3e9553-733b-4bcf-bb90-e20d37ba7efb)\"" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podUID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb"
Sep 17 20:51:19.877356 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.877317    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"]
Sep 17 20:51:19.880017 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.880001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"
Sep 17 20:51:19.883321 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.883302    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 20:51:19.883608 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.883591    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fs7pr\""
Sep 17 20:51:19.885034 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.885002    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 20:51:19.891664 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.891641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"]
Sep 17 20:51:19.977999 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:19.977964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxnd\" (UniqueName: \"kubernetes.io/projected/a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec-kube-api-access-rtxnd\") pod \"migrator-76794cbd6c-mxr2k\" (UID: \"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"
Sep 17 20:51:20.078946 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:20.078912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxnd\" (UniqueName: \"kubernetes.io/projected/a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec-kube-api-access-rtxnd\") pod \"migrator-76794cbd6c-mxr2k\" (UID: \"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"
Sep 17 20:51:20.087505 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:20.087477    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxnd\" (UniqueName: \"kubernetes.io/projected/a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec-kube-api-access-rtxnd\") pod \"migrator-76794cbd6c-mxr2k\" (UID: \"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"
Sep 17 20:51:20.189206 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:20.189115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"
Sep 17 20:51:20.456509 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:20.456428    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k"]
Sep 17 20:51:20.459557 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:20.459526    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4caaf28_5fa8_40c0_b139_15cb2c1ad9ec.slice/crio-906626e850fbcab3c1920719db1df9783160a0f49c06013d3b65a31f25374e20 WatchSource:0}: Error finding container 906626e850fbcab3c1920719db1df9783160a0f49c06013d3b65a31f25374e20: Status 404 returned error can't find the container with id 906626e850fbcab3c1920719db1df9783160a0f49c06013d3b65a31f25374e20
Sep 17 20:51:21.103244 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:21.103203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k" event={"ID":"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec","Type":"ContainerStarted","Data":"906626e850fbcab3c1920719db1df9783160a0f49c06013d3b65a31f25374e20"}
Sep 17 20:51:21.287861 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:21.287806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:51:21.288038 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:21.287977    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:51:21.288175 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:21.288053    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs podName:5e9414f0-9f8a-4d7c-bc9d-527f06ff7927 nodeName:}" failed. No retries permitted until 2026-09-17 20:53:23.288032799 +0000 UTC m=+253.867268641 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs") pod "network-metrics-daemon-77s9j" (UID: "5e9414f0-9f8a-4d7c-bc9d-527f06ff7927") : secret "metrics-daemon-secret" not found
Sep 17 20:51:22.108237 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.108203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k" event={"ID":"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec","Type":"ContainerStarted","Data":"b9677dd48f98b38b274148bed0e9c484269ca86704dcfd982de5af3088e30292"}
Sep 17 20:51:22.108610 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.108253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k" event={"ID":"a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec","Type":"ContainerStarted","Data":"438e31286396f3ddadcd050ff929ca43e84c611024bf4e8ed50e67422a2c6bee"}
Sep 17 20:51:22.132495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.132447    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-mxr2k" podStartSLOduration=2.294121572 podStartE2EDuration="3.132433313s" totalImagesPullingTime="838.311741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:22.132174678 +0000 UTC m=+132.711410528" watchObservedRunningTime="2026-09-17 20:51:22.132433313 +0000 UTC m=+132.711669162"
Sep 17 20:51:22.176687 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.176657    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kns7q"]
Sep 17 20:51:22.178550 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.178536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.180769 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.180743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 20:51:22.180924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.180907    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 20:51:22.181165 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.181143    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 20:51:22.181257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.181185    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 20:51:22.181554 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.181542    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-75xvc\""
Sep 17 20:51:22.194342 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.194322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kns7q"]
Sep 17 20:51:22.295463 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.295428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-cabundle\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.295463 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.295461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5474v\" (UniqueName: \"kubernetes.io/projected/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-kube-api-access-5474v\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.295683 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.295498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-key\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.396221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.396141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-cabundle\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.396221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.396183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5474v\" (UniqueName: \"kubernetes.io/projected/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-kube-api-access-5474v\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.396221 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.396214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-key\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.396837 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.396818    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-cabundle\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.398662 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.398641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-signing-key\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.407126 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.407081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5474v\" (UniqueName: \"kubernetes.io/projected/1ee24b50-10f2-419f-9bcd-19cf1e698ad5-kube-api-access-5474v\") pod \"service-ca-55f6fc7b86-kns7q\" (UID: \"1ee24b50-10f2-419f-9bcd-19cf1e698ad5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.487019 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.486983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q"
Sep 17 20:51:22.729768 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.729666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:22.729768 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.729739    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:22.730090 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.730077    2582 scope.go:117] "RemoveContainer" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"
Sep 17 20:51:22.730273 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:22.730254    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-bdpxp_openshift-console-operator(4f3e9553-733b-4bcf-bb90-e20d37ba7efb)\"" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podUID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb"
Sep 17 20:51:22.755484 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:22.755454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kns7q"]
Sep 17 20:51:22.758286 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:22.758263    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee24b50_10f2_419f_9bcd_19cf1e698ad5.slice/crio-84b64e37cbc5eb8842349007bafd1ae7d101c3cfc0ab2d6890bc1549429e3db4 WatchSource:0}: Error finding container 84b64e37cbc5eb8842349007bafd1ae7d101c3cfc0ab2d6890bc1549429e3db4: Status 404 returned error can't find the container with id 84b64e37cbc5eb8842349007bafd1ae7d101c3cfc0ab2d6890bc1549429e3db4
Sep 17 20:51:23.112577 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:23.112542    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q" event={"ID":"1ee24b50-10f2-419f-9bcd-19cf1e698ad5","Type":"ContainerStarted","Data":"84b64e37cbc5eb8842349007bafd1ae7d101c3cfc0ab2d6890bc1549429e3db4"}
Sep 17 20:51:24.917834 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:24.917799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:24.917834 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:24.917837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:24.918258 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:24.917983    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.918258 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:24.918042    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls podName:0e7e13d1-3c12-48bf-b0a2-4212c7d05df2 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:40.918027228 +0000 UTC m=+151.497263058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cnjp" (UID: "0e7e13d1-3c12-48bf-b0a2-4212c7d05df2") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:24.918350 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:24.918330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:25.120296 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:25.120262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q" event={"ID":"1ee24b50-10f2-419f-9bcd-19cf1e698ad5","Type":"ContainerStarted","Data":"15bbccec4827127d75eb8a6624e1530ea45e6d457f271910e9a00c21307e2546"}
Sep 17 20:51:25.138584 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:25.138533    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-kns7q" podStartSLOduration=1.5836999029999999 podStartE2EDuration="3.138501649s" totalImagesPullingTime="1.554801746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:25.137675721 +0000 UTC m=+135.716911575" watchObservedRunningTime="2026-09-17 20:51:25.138501649 +0000 UTC m=+135.717737498"
Sep 17 20:51:33.578446 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:33.578409    2582 scope.go:117] "RemoveContainer" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"
Sep 17 20:51:34.156003 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.155976    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:51:34.156425 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.156409    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/1.log"
Sep 17 20:51:34.156475 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.156444    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb" containerID="d55b7020dafe6decf431188e14df5f47eccc04ad266e2e0a7c69e34544c9915f" exitCode=255
Sep 17 20:51:34.156513 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.156479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" event={"ID":"4f3e9553-733b-4bcf-bb90-e20d37ba7efb","Type":"ContainerDied","Data":"d55b7020dafe6decf431188e14df5f47eccc04ad266e2e0a7c69e34544c9915f"}
Sep 17 20:51:34.156549 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.156522    2582 scope.go:117] "RemoveContainer" containerID="45e2048104cf60b856b21095067c368eece75276ba149db6b5c0e86a217b6f21"
Sep 17 20:51:34.156813 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:34.156790    2582 scope.go:117] "RemoveContainer" containerID="d55b7020dafe6decf431188e14df5f47eccc04ad266e2e0a7c69e34544c9915f"
Sep 17 20:51:34.156982 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:34.156964    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-bdpxp_openshift-console-operator(4f3e9553-733b-4bcf-bb90-e20d37ba7efb)\"" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podUID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb"
Sep 17 20:51:35.161497 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:35.161471    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:51:40.940429 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:40.940390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:40.942628 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:40.942603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e7e13d1-3c12-48bf-b0a2-4212c7d05df2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cnjp\" (UID: \"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2\") " pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:41.226917 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:41.226835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p9nvv\""
Sep 17 20:51:41.235009 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:41.234972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cnjp"
Sep 17 20:51:41.502905 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:41.502879    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cnjp"]
Sep 17 20:51:41.505719 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:41.505691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7e13d1_3c12_48bf_b0a2_4212c7d05df2.slice/crio-301feec5bff76e083b81f240951d47fd9d1f8961c6519ea7c1118339e96c97a3 WatchSource:0}: Error finding container 301feec5bff76e083b81f240951d47fd9d1f8961c6519ea7c1118339e96c97a3: Status 404 returned error can't find the container with id 301feec5bff76e083b81f240951d47fd9d1f8961c6519ea7c1118339e96c97a3
Sep 17 20:51:42.185555 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.185525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cnjp" event={"ID":"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2","Type":"ContainerStarted","Data":"61232415adbd048fa7a46e4281157672e8eef7e84f44ca11ceabee3ca9639485"}
Sep 17 20:51:42.185924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.185579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cnjp" event={"ID":"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2","Type":"ContainerStarted","Data":"9cb0a8a218bba234d02ddd054dcc80b6f00cde57eb4f50e5b7383019da0aefc8"}
Sep 17 20:51:42.185924 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.185596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cnjp" event={"ID":"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2","Type":"ContainerStarted","Data":"301feec5bff76e083b81f240951d47fd9d1f8961c6519ea7c1118339e96c97a3"}
Sep 17 20:51:42.729323 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.729275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:42.729504 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.729361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:42.730216 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:42.730188    2582 scope.go:117] "RemoveContainer" containerID="d55b7020dafe6decf431188e14df5f47eccc04ad266e2e0a7c69e34544c9915f"
Sep 17 20:51:42.730506 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:42.730480    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-bdpxp_openshift-console-operator(4f3e9553-733b-4bcf-bb90-e20d37ba7efb)\"" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podUID="4f3e9553-733b-4bcf-bb90-e20d37ba7efb"
Sep 17 20:51:44.194298 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:44.194257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cnjp" event={"ID":"0e7e13d1-3c12-48bf-b0a2-4212c7d05df2","Type":"ContainerStarted","Data":"eb3e2ca938e2460d36b5a3126b3d1b0a46c27555466956e8ed90b70aab338140"}
Sep 17 20:51:44.217917 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:44.217868    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2cnjp" podStartSLOduration=33.556492557 podStartE2EDuration="35.217849689s" totalImagesPullingTime="1.661357132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:44.217349235 +0000 UTC m=+154.796585085" watchObservedRunningTime="2026-09-17 20:51:44.217849689 +0000 UTC m=+154.797085536"
Sep 17 20:51:47.100806 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.100774    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"]
Sep 17 20:51:47.102804 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.102788    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:47.106076 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.106057    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 20:51:47.107066 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.107050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mr9hr\""
Sep 17 20:51:47.120373 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.120349    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"]
Sep 17 20:51:47.183698 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.183660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/869782de-c32c-4d9b-aae6-2157b745bac4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-bqgv4\" (UID: \"869782de-c32c-4d9b-aae6-2157b745bac4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:47.284389 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.284349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/869782de-c32c-4d9b-aae6-2157b745bac4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-bqgv4\" (UID: \"869782de-c32c-4d9b-aae6-2157b745bac4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:47.286786 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.286758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/869782de-c32c-4d9b-aae6-2157b745bac4-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-bqgv4\" (UID: \"869782de-c32c-4d9b-aae6-2157b745bac4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:47.411870 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.411775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:47.693201 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:47.693130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"]
Sep 17 20:51:47.695999 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:47.695965    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869782de_c32c_4d9b_aae6_2157b745bac4.slice/crio-a12c52849be5197a0b51b75540397053ea32316a362a3626416d0e937698efc4 WatchSource:0}: Error finding container a12c52849be5197a0b51b75540397053ea32316a362a3626416d0e937698efc4: Status 404 returned error can't find the container with id a12c52849be5197a0b51b75540397053ea32316a362a3626416d0e937698efc4
Sep 17 20:51:48.032648 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:48.032610    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8hlpm" podUID="4874db31-3321-40bf-8eed-2b111959c3ac"
Sep 17 20:51:48.211903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:48.211870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4" event={"ID":"869782de-c32c-4d9b-aae6-2157b745bac4","Type":"ContainerStarted","Data":"a12c52849be5197a0b51b75540397053ea32316a362a3626416d0e937698efc4"}
Sep 17 20:51:48.211903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:48.211910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:51:49.216297 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:49.216266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4" event={"ID":"869782de-c32c-4d9b-aae6-2157b745bac4","Type":"ContainerStarted","Data":"e604d93b536b85cc0c8de5d54d7306a51a9074e4dfda65e372d56bac84c1c467"}
Sep 17 20:51:49.216658 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:49.216474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:49.220977 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:49.220953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4"
Sep 17 20:51:49.235538 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:49.235493    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-bqgv4" podStartSLOduration=1.184671108 podStartE2EDuration="2.23547971s" totalImagesPullingTime="1.050808602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:49.234998278 +0000 UTC m=+159.814234128" watchObservedRunningTime="2026-09-17 20:51:49.23547971 +0000 UTC m=+159.814715560"
Sep 17 20:51:49.602805 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:49.602766    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-77s9j" podUID="5e9414f0-9f8a-4d7c-bc9d-527f06ff7927"
Sep 17 20:51:52.925033 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:52.924983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:51:52.927296 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:52.927277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4874db31-3321-40bf-8eed-2b111959c3ac-metrics-tls\") pod \"dns-default-8hlpm\" (UID: \"4874db31-3321-40bf-8eed-2b111959c3ac\") " pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:51:53.015337 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.015305    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2jjw4\""
Sep 17 20:51:53.023360 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.023345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:51:53.294704 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.294655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8hlpm"]
Sep 17 20:51:53.297499 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:53.297473    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4874db31_3321_40bf_8eed_2b111959c3ac.slice/crio-6bd7f3843e46fb28836f6bf60b88e3286615857a17cf9b6241a50d6684f9bb70 WatchSource:0}: Error finding container 6bd7f3843e46fb28836f6bf60b88e3286615857a17cf9b6241a50d6684f9bb70: Status 404 returned error can't find the container with id 6bd7f3843e46fb28836f6bf60b88e3286615857a17cf9b6241a50d6684f9bb70
Sep 17 20:51:53.669775 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.669696    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"]
Sep 17 20:51:53.674377 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.674348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.679229 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.678789    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 20:51:53.679229 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.678895    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:51:53.679229 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.678942    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:51:53.679229 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.679036    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:51:53.679229 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.679231    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4dsf8\""
Sep 17 20:51:53.681280 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.681177    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 20:51:53.696998 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.696970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"]
Sep 17 20:51:53.697142 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.697107    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zfgl7"]
Sep 17 20:51:53.703394 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.703372    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.707644 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.707623    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:51:53.708179 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.707875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:51:53.708179 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.707999    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:51:53.708179 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.708070    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8f996\""
Sep 17 20:51:53.730146 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd8b3fe-a405-4588-bdca-7237c11763b1-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.730275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730178    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-wtmp\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-sys\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730429 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730429 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wxm\" (UniqueName: \"kubernetes.io/projected/bbd8b3fe-a405-4588-bdca-7237c11763b1-kube-api-access-67wxm\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.730429 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-root\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730554 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-textfile\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730554 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-metrics-client-ca\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730554 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.730714 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.730714 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730638    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.730810 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.730717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csgvm\" (UniqueName: \"kubernetes.io/projected/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-kube-api-access-csgvm\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831336 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-textfile\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831336 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-metrics-client-ca\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.831568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.831711 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csgvm\" (UniqueName: \"kubernetes.io/projected/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-kube-api-access-csgvm\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831711 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd8b3fe-a405-4588-bdca-7237c11763b1-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.831711 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-wtmp\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831711 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-textfile\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-sys\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-sys\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wxm\" (UniqueName: \"kubernetes.io/projected/bbd8b3fe-a405-4588-bdca-7237c11763b1-kube-api-access-67wxm\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.831912 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-root\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-wtmp\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.831966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-root\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.832040    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:53.832045    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.832083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-metrics-client-ca\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.832236 ip-10-0-131-59 kubenswrapper[2582]: E0917 20:51:53.832142    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls podName:14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e nodeName:}" failed. No retries permitted until 2026-09-17 20:51:54.332120159 +0000 UTC m=+164.911355991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls") pod "node-exporter-zfgl7" (UID: "14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e") : secret "node-exporter-tls" not found
Sep 17 20:51:53.832508 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.832481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbd8b3fe-a405-4588-bdca-7237c11763b1-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.834342 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.834319    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.834454 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.834417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.834529 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.834512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbd8b3fe-a405-4588-bdca-7237c11763b1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.844732 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.844709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wxm\" (UniqueName: \"kubernetes.io/projected/bbd8b3fe-a405-4588-bdca-7237c11763b1-kube-api-access-67wxm\") pod \"openshift-state-metrics-86c5bc79d6-mbxjw\" (UID: \"bbd8b3fe-a405-4588-bdca-7237c11763b1\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:53.848938 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.848920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csgvm\" (UniqueName: \"kubernetes.io/projected/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-kube-api-access-csgvm\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:53.987324 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:53.987251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"
Sep 17 20:51:54.234165 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:54.234125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hlpm" event={"ID":"4874db31-3321-40bf-8eed-2b111959c3ac","Type":"ContainerStarted","Data":"6bd7f3843e46fb28836f6bf60b88e3286615857a17cf9b6241a50d6684f9bb70"}
Sep 17 20:51:54.280986 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:54.280761    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw"]
Sep 17 20:51:54.283236 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:54.283207    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd8b3fe_a405_4588_bdca_7237c11763b1.slice/crio-e85cdf76a7b05eb11083bf95d57f5feff42d371c8fe251a9356db2e68dc7b4ba WatchSource:0}: Error finding container e85cdf76a7b05eb11083bf95d57f5feff42d371c8fe251a9356db2e68dc7b4ba: Status 404 returned error can't find the container with id e85cdf76a7b05eb11083bf95d57f5feff42d371c8fe251a9356db2e68dc7b4ba
Sep 17 20:51:54.335335 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:54.335308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:54.337522 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:54.337502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e-node-exporter-tls\") pod \"node-exporter-zfgl7\" (UID: \"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e\") " pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:54.616257 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:54.616155    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zfgl7"
Sep 17 20:51:54.920537 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:54.920498    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c2cacc_4fc5_4ef8_ba48_ad9f121f3f9e.slice/crio-6b607e5080e7b0d040affa82a18245c234e721cefcd11c6ab8c3480caad8540a WatchSource:0}: Error finding container 6b607e5080e7b0d040affa82a18245c234e721cefcd11c6ab8c3480caad8540a: Status 404 returned error can't find the container with id 6b607e5080e7b0d040affa82a18245c234e721cefcd11c6ab8c3480caad8540a
Sep 17 20:51:55.240185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.240055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw" event={"ID":"bbd8b3fe-a405-4588-bdca-7237c11763b1","Type":"ContainerStarted","Data":"108636e9d3c643d65d9fe043c0bff0d55cd66ea07b21a03610a1dadc31eb9636"}
Sep 17 20:51:55.240185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.240147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw" event={"ID":"bbd8b3fe-a405-4588-bdca-7237c11763b1","Type":"ContainerStarted","Data":"7e92f479b09962fe71a31dcb49e176d47ddefc638eefabf17eae8ec0565fd028"}
Sep 17 20:51:55.240185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.240178    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw" event={"ID":"bbd8b3fe-a405-4588-bdca-7237c11763b1","Type":"ContainerStarted","Data":"e85cdf76a7b05eb11083bf95d57f5feff42d371c8fe251a9356db2e68dc7b4ba"}
Sep 17 20:51:55.242105 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.242058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hlpm" event={"ID":"4874db31-3321-40bf-8eed-2b111959c3ac","Type":"ContainerStarted","Data":"551d4b0a4fff09a7f3341154a20468d2d1de7d1fd7e393179be103c4bff2bd1d"}
Sep 17 20:51:55.242224 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.242133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8hlpm" event={"ID":"4874db31-3321-40bf-8eed-2b111959c3ac","Type":"ContainerStarted","Data":"0218c735aaeaa78d3f1f363f3d9e7b0069a12e84db5afde15b7c97c0455b6f96"}
Sep 17 20:51:55.242224 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.242164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:51:55.243334 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.243311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfgl7" event={"ID":"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e","Type":"ContainerStarted","Data":"6b607e5080e7b0d040affa82a18245c234e721cefcd11c6ab8c3480caad8540a"}
Sep 17 20:51:55.263490 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.263446    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8hlpm" podStartSLOduration=129.600697111 podStartE2EDuration="2m11.263430411s" totalImagesPullingTime="1.6627333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:55.262405014 +0000 UTC m=+165.841640865" watchObservedRunningTime="2026-09-17 20:51:55.263430411 +0000 UTC m=+165.842666263"
Sep 17 20:51:55.578240 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:55.578211    2582 scope.go:117] "RemoveContainer" containerID="d55b7020dafe6decf431188e14df5f47eccc04ad266e2e0a7c69e34544c9915f"
Sep 17 20:51:56.249444 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.249410    2582 generic.go:356] "Generic (PLEG): container finished" podID="14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e" containerID="b4d3f6f85e3041fb9ee07edd3e29e79d04b738e16cee26181016e78e768553d4" exitCode=0
Sep 17 20:51:56.249883 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.249497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfgl7" event={"ID":"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e","Type":"ContainerDied","Data":"b4d3f6f85e3041fb9ee07edd3e29e79d04b738e16cee26181016e78e768553d4"}
Sep 17 20:51:56.251632 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.251611    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:51:56.251746 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.251692    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" event={"ID":"4f3e9553-733b-4bcf-bb90-e20d37ba7efb","Type":"ContainerStarted","Data":"07fe66dd479a749d63f09d6c815d5306dfb1c3b49dea30135421ffd5e61d92fd"}
Sep 17 20:51:56.255130 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.252433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:56.258984 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.258902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw" event={"ID":"bbd8b3fe-a405-4588-bdca-7237c11763b1","Type":"ContainerStarted","Data":"2c05ec803bea2cb78eb499e2d4e80a9b082ad1e7bdfd5687257c4a2452a4c978"}
Sep 17 20:51:56.262258 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.262239    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp"
Sep 17 20:51:56.368578 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.368549    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-5thql"]
Sep 17 20:51:56.373344 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.373047    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:51:56.373344 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.373146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-mbxjw" podStartSLOduration=2.338791999 podStartE2EDuration="3.373127427s" totalImagesPullingTime="1.034335428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:56.371788403 +0000 UTC m=+166.951024255" watchObservedRunningTime="2026-09-17 20:51:56.373127427 +0000 UTC m=+166.952363279"
Sep 17 20:51:56.376156 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.376139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 20:51:56.376493 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.376472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 20:51:56.376571 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.376556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-czmcg\""
Sep 17 20:51:56.396057 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.396034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5thql"]
Sep 17 20:51:56.410268 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.410215    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-bdpxp" podStartSLOduration=42.402831922 podStartE2EDuration="44.410196928s" totalImagesPullingTime="2.007365006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:56.410046309 +0000 UTC m=+166.989282152" watchObservedRunningTime="2026-09-17 20:51:56.410196928 +0000 UTC m=+166.989432780"
Sep 17 20:51:56.453041 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.453013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkv9\" (UniqueName: \"kubernetes.io/projected/84f3a2ea-e3ea-449c-a292-7388c76c730a-kube-api-access-2bkv9\") pod \"downloads-6594f4c75-5thql\" (UID: \"84f3a2ea-e3ea-449c-a292-7388c76c730a\") " pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:51:56.554466 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.554380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkv9\" (UniqueName: \"kubernetes.io/projected/84f3a2ea-e3ea-449c-a292-7388c76c730a-kube-api-access-2bkv9\") pod \"downloads-6594f4c75-5thql\" (UID: \"84f3a2ea-e3ea-449c-a292-7388c76c730a\") " pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:51:56.568752 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.568724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkv9\" (UniqueName: \"kubernetes.io/projected/84f3a2ea-e3ea-449c-a292-7388c76c730a-kube-api-access-2bkv9\") pod \"downloads-6594f4c75-5thql\" (UID: \"84f3a2ea-e3ea-449c-a292-7388c76c730a\") " pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:51:56.680166 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.680133    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"]
Sep 17 20:51:56.682652 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.682622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:51:56.683796 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.683778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.687735 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.687709    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 20:51:56.688349 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688322    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 20:51:56.688349 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 20:51:56.688481 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 20:51:56.688541 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688510    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 20:51:56.688776 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-a331s82vjaqre\""
Sep 17 20:51:56.688836 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.688826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fcv6t\""
Sep 17 20:51:56.708202 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.708171    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"]
Sep 17 20:51:56.756800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.756975 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgw2\" (UniqueName: \"kubernetes.io/projected/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-kube-api-access-2sgw2\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.756975 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-metrics-client-ca\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.756975 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-grpc-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.756975 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.756975 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.757243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.756996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.757243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.757042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-metrics-client-ca\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-grpc-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857582 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.857903 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.857731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgw2\" (UniqueName: \"kubernetes.io/projected/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-kube-api-access-2sgw2\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.858763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.858708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-metrics-client-ca\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860477 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860615 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-grpc-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860615 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-tls\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860615 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860715 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.860715 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.860655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.881316 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.881289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgw2\" (UniqueName: \"kubernetes.io/projected/e5e784c6-209f-47c5-8bd0-c70fe73d6ab9-kube-api-access-2sgw2\") pod \"thanos-querier-78ccfc565f-sbzqf\" (UID: \"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9\") " pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:56.961588 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.961560    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5thql"]
Sep 17 20:51:56.964359 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:56.964333    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f3a2ea_e3ea_449c_a292_7388c76c730a.slice/crio-114eeda88bbb916d7a27f0c4dc4610862b977db8fe44c78ef0d78d871d30e430 WatchSource:0}: Error finding container 114eeda88bbb916d7a27f0c4dc4610862b977db8fe44c78ef0d78d871d30e430: Status 404 returned error can't find the container with id 114eeda88bbb916d7a27f0c4dc4610862b977db8fe44c78ef0d78d871d30e430
Sep 17 20:51:56.995286 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:56.995247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:51:57.265529 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:57.265497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfgl7" event={"ID":"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e","Type":"ContainerStarted","Data":"b623fed89c6e0db803c4c9b14a490fcd352894e7b4dd6da16ff632c4d42ab23c"}
Sep 17 20:51:57.265972 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:57.265556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zfgl7" event={"ID":"14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e","Type":"ContainerStarted","Data":"8eba6320437e29f5ab6ecf086350e604e24575fa7e4b22ef89bf503768abeaac"}
Sep 17 20:51:57.266700 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:57.266671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5thql" event={"ID":"84f3a2ea-e3ea-449c-a292-7388c76c730a","Type":"ContainerStarted","Data":"114eeda88bbb916d7a27f0c4dc4610862b977db8fe44c78ef0d78d871d30e430"}
Sep 17 20:51:57.271568 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:57.271550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"]
Sep 17 20:51:57.273660 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:51:57.273635    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e784c6_209f_47c5_8bd0_c70fe73d6ab9.slice/crio-1f600d98ad81f6bd0d5362026ffe51f4073582bca71dd581f74828d046a5ed73 WatchSource:0}: Error finding container 1f600d98ad81f6bd0d5362026ffe51f4073582bca71dd581f74828d046a5ed73: Status 404 returned error can't find the container with id 1f600d98ad81f6bd0d5362026ffe51f4073582bca71dd581f74828d046a5ed73
Sep 17 20:51:57.288021 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:57.287980    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zfgl7" podStartSLOduration=3.179993522 podStartE2EDuration="4.28796438s" totalImagesPullingTime="1.047831318s" totalInitContainerRuntime="60.13954ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:57.286647123 +0000 UTC m=+167.865882972" watchObservedRunningTime="2026-09-17 20:51:57.28796438 +0000 UTC m=+167.867200231"
Sep 17 20:51:58.273219 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:58.273117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"1f600d98ad81f6bd0d5362026ffe51f4073582bca71dd581f74828d046a5ed73"}
Sep 17 20:51:59.284185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:59.284115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"649ee8b738fe08e00214cdb5c8c5297d67b313bc4d8370bf2a560dbae6a9a5d6"}
Sep 17 20:51:59.284599 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:51:59.284274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"a2da0e97643d48c434b12db46d02d5e6dff3131ad38c246470c82d4afec94de1"}
Sep 17 20:52:00.050944 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.050616    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:52:00.054561 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.054542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.057593 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.057569    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 20:52:00.057718 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.057654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 20:52:00.058153 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058132    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 20:52:00.058284 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058234    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 20:52:00.058503 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 20:52:00.058599 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 20:52:00.058599 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058484    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 20:52:00.058732 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058706    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 20:52:00.058732 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058713    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftxd9\""
Sep 17 20:52:00.059081 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.058944    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 20:52:00.060973 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.060953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 20:52:00.061505 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.061488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 20:52:00.061937 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.061920    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 20:52:00.062171 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.062157    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 20:52:00.062452 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.062436    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ehvkaie441deb\""
Sep 17 20:52:00.064406 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.064376    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 20:52:00.067596 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.067579    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 20:52:00.086375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086495 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpl2r\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.086736 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.086978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.087018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.087007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.089631 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.089610    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:52:00.188438 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188408    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188629 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188960 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188960 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl2r\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.188960 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.188952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.189684 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.189555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.190382 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.190042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.192243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.190707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.192243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.191380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.192243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.191794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.192243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.192142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.192243 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.192196    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.193887 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.193865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.194722 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.194658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.194821 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.194762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.195275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.195249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.196214 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.196171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.196303 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.196213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.196360 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.196324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.196585 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.196562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.198744 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.198723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.210409 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.210356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl2r\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r\") pod \"prometheus-k8s-0\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.291889 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.291854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"35e4f8e742b3bbaa52f5c07c5425b7f56271eee884a4d791df0042942dd6ed99"}
Sep 17 20:52:00.292375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.291919    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"e9a69e41b825347942d1d1b946d698e5d1e199a9e07e07fb32773ad795405c4d"}
Sep 17 20:52:00.292375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.291959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"002181736cfca50c20793b3763f714e0d0930e94566f35a4812eb9b9225eca90"}
Sep 17 20:52:00.292375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.291987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" event={"ID":"e5e784c6-209f-47c5-8bd0-c70fe73d6ab9","Type":"ContainerStarted","Data":"44dac1f2c320ff5630c4e80a9860d912ae00740fbe83ce7c020602d38aa6440a"}
Sep 17 20:52:00.292375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.292027    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:52:00.319614 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.319559    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf" podStartSLOduration=1.869676637 podStartE2EDuration="4.3195416s" totalImagesPullingTime="2.449864963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:00.317493735 +0000 UTC m=+170.896729585" watchObservedRunningTime="2026-09-17 20:52:00.3195416 +0000 UTC m=+170.898777451"
Sep 17 20:52:00.380385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.380346    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:00.578110 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.578073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:52:00.680161 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:00.680138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:52:00.682209 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:52:00.682180    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c39bc7_191a_4f0d_8f94_8e6376ed125c.slice/crio-80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe WatchSource:0}: Error finding container 80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe: Status 404 returned error can't find the container with id 80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe
Sep 17 20:52:01.298613 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:01.298573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe"}
Sep 17 20:52:02.304876 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:02.304838    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="322ce91fd5612997909127c0586b1b99e7b8f749daa738cbdc5c7c7f00098c6b" exitCode=0
Sep 17 20:52:02.305378 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:02.304927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"322ce91fd5612997909127c0586b1b99e7b8f749daa738cbdc5c7c7f00098c6b"}
Sep 17 20:52:05.266057 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:05.266020    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8hlpm"
Sep 17 20:52:06.307422 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.307398    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78ccfc565f-sbzqf"
Sep 17 20:52:06.329041 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"cd1a15d10ce0c07d93981e7746d113a6734cd4ce864d05db582aa1a1e2d35053"}
Sep 17 20:52:06.329227 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"834a23c1ada68b0fd4a9d4e606a72c7872e3b670deca095a434fe9c544f019a7"}
Sep 17 20:52:06.329227 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"892e400c60f492c38e7584d5ac51bc46ae8ca0bad0fa7e2a0b1326847c098872"}
Sep 17 20:52:06.329227 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"865fe520027b4184895cd019432c42c2fe9954770d83915766ac374463000a4f"}
Sep 17 20:52:06.329380 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"4bd382b421d000c29186ee492dea84fe12857ca3957691f5acaa98e0ca1e4dca"}
Sep 17 20:52:06.329380 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.329276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerStarted","Data":"8d40636f3c99778c2c3c332a49e09acd2c3cd83249af1d17a563aa015051a903"}
Sep 17 20:52:06.371587 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:06.371537    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.9322562159999999 podStartE2EDuration="6.371519306s" totalImagesPullingTime="4.417778279s" totalInitContainerRuntime="21.484811ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:52:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:06.369904354 +0000 UTC m=+176.949140218" watchObservedRunningTime="2026-09-17 20:52:06.371519306 +0000 UTC m=+176.950755157"
Sep 17 20:52:10.380687 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:10.380647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:52:14.366022 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:14.365976    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5thql" event={"ID":"84f3a2ea-e3ea-449c-a292-7388c76c730a","Type":"ContainerStarted","Data":"b7b7fe9263057d090c6cc3fa41b4d4882ab4bafe89383ff87a33326650f2941e"}
Sep 17 20:52:14.366527 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:14.366151    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:52:14.386454 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:14.386421    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-5thql"
Sep 17 20:52:14.387972 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:14.387927    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-5thql" podStartSLOduration=1.818157147 podStartE2EDuration="18.387911795s" totalImagesPullingTime="16.569754648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:52:14.386659918 +0000 UTC m=+184.965895752" watchObservedRunningTime="2026-09-17 20:52:14.387911795 +0000 UTC m=+184.967147678"
Sep 17 20:52:15.725236 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:15.725207    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hlpm_4874db31-3321-40bf-8eed-2b111959c3ac/dns/0.log"
Sep 17 20:52:15.927021 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:15.926990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hlpm_4874db31-3321-40bf-8eed-2b111959c3ac/kube-rbac-proxy/0.log"
Sep 17 20:52:17.126281 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:17.126256    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64s7w_3549c13d-d5d4-4694-bfe1-f4b0d1d069af/dns-node-resolver/0.log"
Sep 17 20:52:17.727375 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:52:17.727346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bsjht_445ed9f2-bb49-4023-ad4c-ef2b81082509/serve-healthcheck-canary/0.log"
Sep 17 20:53:00.380822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:00.380779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:00.439265 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:00.439238    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:00.572561 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:00.572532    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:18.393469 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393438    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393739    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="prometheus" containerID="cri-o://8d40636f3c99778c2c3c332a49e09acd2c3cd83249af1d17a563aa015051a903" gracePeriod=600
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393793    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cd1a15d10ce0c07d93981e7746d113a6734cd4ce864d05db582aa1a1e2d35053" gracePeriod=600
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393793    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="thanos-sidecar" containerID="cri-o://865fe520027b4184895cd019432c42c2fe9954770d83915766ac374463000a4f" gracePeriod=600
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393755    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy" containerID="cri-o://834a23c1ada68b0fd4a9d4e606a72c7872e3b670deca095a434fe9c544f019a7" gracePeriod=600
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393832    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-web" containerID="cri-o://892e400c60f492c38e7584d5ac51bc46ae8ca0bad0fa7e2a0b1326847c098872" gracePeriod=600
Sep 17 20:53:18.393982 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.393793    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="config-reloader" containerID="cri-o://4bd382b421d000c29186ee492dea84fe12857ca3957691f5acaa98e0ca1e4dca" gracePeriod=600
Sep 17 20:53:18.630759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630732    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="cd1a15d10ce0c07d93981e7746d113a6734cd4ce864d05db582aa1a1e2d35053" exitCode=0
Sep 17 20:53:18.630759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630755    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="834a23c1ada68b0fd4a9d4e606a72c7872e3b670deca095a434fe9c544f019a7" exitCode=0
Sep 17 20:53:18.630759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"cd1a15d10ce0c07d93981e7746d113a6734cd4ce864d05db582aa1a1e2d35053"}
Sep 17 20:53:18.630759 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630739    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"834a23c1ada68b0fd4a9d4e606a72c7872e3b670deca095a434fe9c544f019a7"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"892e400c60f492c38e7584d5ac51bc46ae8ca0bad0fa7e2a0b1326847c098872"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630765    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="892e400c60f492c38e7584d5ac51bc46ae8ca0bad0fa7e2a0b1326847c098872" exitCode=0
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630865    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"865fe520027b4184895cd019432c42c2fe9954770d83915766ac374463000a4f"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630862    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="865fe520027b4184895cd019432c42c2fe9954770d83915766ac374463000a4f" exitCode=0
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630880    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="4bd382b421d000c29186ee492dea84fe12857ca3957691f5acaa98e0ca1e4dca" exitCode=0
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630886    2582 generic.go:356] "Generic (PLEG): container finished" podID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerID="8d40636f3c99778c2c3c332a49e09acd2c3cd83249af1d17a563aa015051a903" exitCode=0
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"4bd382b421d000c29186ee492dea84fe12857ca3957691f5acaa98e0ca1e4dca"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"8d40636f3c99778c2c3c332a49e09acd2c3cd83249af1d17a563aa015051a903"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b9c39bc7-191a-4f0d-8f94-8e6376ed125c","Type":"ContainerDied","Data":"80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe"}
Sep 17 20:53:18.631050 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.630969    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bbc33c64f6a7129b4a2b96a2d9c616afe8f4c598cfc65ee8c88a2d6ce4b5fe"
Sep 17 20:53:18.687803 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687728    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.687803 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.687803 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687822    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpl2r\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687882    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687908    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.687984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688110    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688210    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.688763 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.688443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689671    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689821    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689853    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689880    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689914    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.689951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db\") pod \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\" (UID: \"b9c39bc7-191a-4f0d-8f94-8e6376ed125c\") "
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.690236    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.690255    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.690269    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.690423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.690284    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.691530 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.690768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.691530 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.691122    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.691732 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.691707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.691808 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.691782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.691997 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.691974    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.692067 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692022    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:53:18.692067 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692038    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.692310 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692286    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:18.692379 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692323    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r" (OuterVolumeSpecName: "kube-api-access-bpl2r") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "kube-api-access-bpl2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:18.692716 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:53:18.692943 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.692919    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.693275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.693250    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config" (OuterVolumeSpecName: "config") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.694153 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.694130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.694451 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.694434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.694544 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.694490    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out" (OuterVolumeSpecName: "config-out") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:53:18.704808 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.704782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config" (OuterVolumeSpecName: "web-config") pod "b9c39bc7-191a-4f0d-8f94-8e6376ed125c" (UID: "b9c39bc7-191a-4f0d-8f94-8e6376ed125c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:53:18.791493 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791458    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791493 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791492    2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-grpc-tls\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791508    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791521    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-web-config\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791534    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config-out\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791547    2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791561    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791577    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791591    2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-prometheus-k8s-db\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791604    2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-metrics-client-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791617    2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-kube-rbac-proxy\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791631    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpl2r\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-kube-api-access-bpl2r\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791644    2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791657    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-tls-assets\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.791671 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791670    2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-configmap-metrics-client-ca\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:18.792032 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:18.791682    2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b9c39bc7-191a-4f0d-8f94-8e6376ed125c-config\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 20:53:19.634850 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.634823    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.655515 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.655485    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:19.659712 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.659642    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:19.686382 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686356    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:19.686666 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686653    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-web"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686668    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-web"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686686    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="init-config-reloader"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686693    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="init-config-reloader"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686703    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="prometheus"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686708    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="prometheus"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686716    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="thanos-sidecar"
Sep 17 20:53:19.686725 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686722    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="thanos-sidecar"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686734    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-thanos"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686740    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-thanos"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686759    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="config-reloader"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686765    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="config-reloader"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686771    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686776    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686820    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="config-reloader"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686826    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686832    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="thanos-sidecar"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686838    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="prometheus"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686843    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-web"
Sep 17 20:53:19.686939 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.686850    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" containerName="kube-rbac-proxy-thanos"
Sep 17 20:53:19.689942 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.689927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.693398 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693376    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 20:53:19.693544 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693430    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 20:53:19.693544 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 20:53:19.693544 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693502    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ehvkaie441deb\""
Sep 17 20:53:19.693710 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693576    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 20:53:19.693754 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.693740    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 20:53:19.694071 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.694049    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 20:53:19.694193 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.694111    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 20:53:19.694193 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.694133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 20:53:19.694305 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.694217    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 20:53:19.694906 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.694891    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 20:53:19.695701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.695239    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 20:53:19.695701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.695337    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftxd9\""
Sep 17 20:53:19.695701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.695413    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 20:53:19.695701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.695578    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 20:53:19.696748 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.696696    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 20:53:19.699145 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.699128    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 20:53:19.708015 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.707876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:19.799450 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799450 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-config-out\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799640    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtf7\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-kube-api-access-9jtf7\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799857    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-web-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.799987 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.799915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901086 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-web-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901086 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901265 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901220    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901265 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901398 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901466 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901525 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901525 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901627 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-config-out\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901627 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901627 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901627 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.901814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901794    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.902062 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.902062 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.902062 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.901910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtf7\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-kube-api-access-9jtf7\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.902062 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.902010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.902062 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.902054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.903800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.902646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.903800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.902791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.903800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.903136    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.903800 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.903476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.904194 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.904002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.905269 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.905225    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1966f40c-d5a1-450c-89b6-139c0dc8f098-config-out\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.905645 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.905624    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1966f40c-d5a1-450c-89b6-139c0dc8f098-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.905752 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.905712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-web-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906739 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.906776 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.906739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.907533 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.907517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-config\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.908194 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.908177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1966f40c-d5a1-450c-89b6-139c0dc8f098-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.911920 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.911879    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtf7\" (UniqueName: \"kubernetes.io/projected/1966f40c-d5a1-450c-89b6-139c0dc8f098-kube-api-access-9jtf7\") pod \"prometheus-k8s-0\" (UID: \"1966f40c-d5a1-450c-89b6-139c0dc8f098\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:19.999826 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:19.999786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:20.285688 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:20.285582    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 20:53:20.288384 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:53:20.288359    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1966f40c_d5a1_450c_89b6_139c0dc8f098.slice/crio-5b45631992a5787509b376f8212de8c34abcff4603b82941fb4b49283d85688c WatchSource:0}: Error finding container 5b45631992a5787509b376f8212de8c34abcff4603b82941fb4b49283d85688c: Status 404 returned error can't find the container with id 5b45631992a5787509b376f8212de8c34abcff4603b82941fb4b49283d85688c
Sep 17 20:53:20.640143 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:20.640026    2582 generic.go:356] "Generic (PLEG): container finished" podID="1966f40c-d5a1-450c-89b6-139c0dc8f098" containerID="d376aee735de63f09cfced5aca77ef4c678ffec6d42a668ebd9ccf60d6bf5216" exitCode=0
Sep 17 20:53:20.640143 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:20.640120    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerDied","Data":"d376aee735de63f09cfced5aca77ef4c678ffec6d42a668ebd9ccf60d6bf5216"}
Sep 17 20:53:20.640588 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:20.640213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"5b45631992a5787509b376f8212de8c34abcff4603b82941fb4b49283d85688c"}
Sep 17 20:53:21.584350 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.584319    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c39bc7-191a-4f0d-8f94-8e6376ed125c" path="/var/lib/kubelet/pods/b9c39bc7-191a-4f0d-8f94-8e6376ed125c/volumes"
Sep 17 20:53:21.647022 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.646992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"7c227a914641f904a64306539cd52380fdef56ef6b8a123d02baae58abe707e8"}
Sep 17 20:53:21.647404 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.647057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"4934881f180ff99603fecf0c1ebeb0803e57d3ec8551cb24fd8a955d1d11b703"}
Sep 17 20:53:21.647404 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.647088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"e9daff5974ce0cddd822aa91ec54944f784451267c2554fc8c4081e9faaa14be"}
Sep 17 20:53:21.647404 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.647144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"d9a8201f8e30f2a438002c0027a3d0678f891b57be8d29208d13b441d72bbeee"}
Sep 17 20:53:21.647404 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.647167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"4a39af00a04a63b78d514bc328d4fd14d11bde8616b22ff5e2dc559f35407dca"}
Sep 17 20:53:21.647404 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.647203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1966f40c-d5a1-450c-89b6-139c0dc8f098","Type":"ContainerStarted","Data":"fa229d97581c93a89685f0a45251d163b61156e55a9438523bfc1f9c6a72129f"}
Sep 17 20:53:21.682204 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:21.682154    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.662116395 podStartE2EDuration="2.682137065s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.02067ms" isStatefulPod=false podCreationTimestamp="2026-09-17 20:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:21.680681049 +0000 UTC m=+252.259916933" watchObservedRunningTime="2026-09-17 20:53:21.682137065 +0000 UTC m=+252.261372967"
Sep 17 20:53:23.336898 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:23.336805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:53:23.339146 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:23.339124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e9414f0-9f8a-4d7c-bc9d-527f06ff7927-metrics-certs\") pod \"network-metrics-daemon-77s9j\" (UID: \"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927\") " pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:53:23.381137 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:23.381113    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8j64n\""
Sep 17 20:53:23.389149 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:23.389129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77s9j"
Sep 17 20:53:23.659979 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:23.659911    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-77s9j"]
Sep 17 20:53:23.663130 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:53:23.663088    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9414f0_9f8a_4d7c_bc9d_527f06ff7927.slice/crio-777c707f480ec2e894b4657484a5f17669c1cdbf4027b3ea34cc122a57c13752 WatchSource:0}: Error finding container 777c707f480ec2e894b4657484a5f17669c1cdbf4027b3ea34cc122a57c13752: Status 404 returned error can't find the container with id 777c707f480ec2e894b4657484a5f17669c1cdbf4027b3ea34cc122a57c13752
Sep 17 20:53:24.659906 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:24.659879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77s9j" event={"ID":"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927","Type":"ContainerStarted","Data":"777c707f480ec2e894b4657484a5f17669c1cdbf4027b3ea34cc122a57c13752"}
Sep 17 20:53:25.000945 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:25.000918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:53:25.665771 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:25.665743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77s9j" event={"ID":"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927","Type":"ContainerStarted","Data":"af3ac2affbdcbc3f1a4f0c478038e9cbb35c66a9038fb3db8952c2ea85f24787"}
Sep 17 20:53:25.666147 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:25.665792    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77s9j" event={"ID":"5e9414f0-9f8a-4d7c-bc9d-527f06ff7927","Type":"ContainerStarted","Data":"f9b098791e8fc513430c2892621d48ac7fc3a8b6113e55d14462fce720e04a7b"}
Sep 17 20:53:25.686593 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:53:25.686544    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-77s9j" podStartSLOduration=253.744837073 podStartE2EDuration="4m14.686529625s" totalImagesPullingTime="941.692552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:53:25.685571517 +0000 UTC m=+256.264807369" watchObservedRunningTime="2026-09-17 20:53:25.686529625 +0000 UTC m=+256.265765475"
Sep 17 20:54:11.455513 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:11.455486    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:54:11.460267 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:11.460235    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:54:11.460896 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:11.460874    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:54:11.465896 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:11.465873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:54:11.466858 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:11.466839    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:54:20.000185 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:20.000133    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:54:20.015641 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:20.015608    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:54:20.907588 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:54:20.907558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 20:57:36.816855 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.816819    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdcgh"]
Sep 17 20:57:36.820055 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.820039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:36.823965 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.823945    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 20:57:36.824115 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.823977    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 20:57:36.824672 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.824659    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hc7jm\""
Sep 17 20:57:36.841566 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.841541    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdcgh"]
Sep 17 20:57:36.870565 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.870544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:36.870701 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.870578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmz5j\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-kube-api-access-dmz5j\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:36.971212 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.971178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:36.971388 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:36.971219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmz5j\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-kube-api-access-dmz5j\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:37.002083 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.002045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-bound-sa-token\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:37.002400 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.002383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmz5j\" (UniqueName: \"kubernetes.io/projected/84299a44-671b-4036-a730-695d26737976-kube-api-access-dmz5j\") pod \"cert-manager-55b4d954d5-xdcgh\" (UID: \"84299a44-671b-4036-a730-695d26737976\") " pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:37.129643 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.129532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-xdcgh"
Sep 17 20:57:37.415814 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.415731    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-xdcgh"]
Sep 17 20:57:37.425648 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:57:37.425613    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84299a44_671b_4036_a730_695d26737976.slice/crio-860cb70ef340ffe818faf0a7ca70e665f38229632aee9297ab5ab6a10f57c385 WatchSource:0}: Error finding container 860cb70ef340ffe818faf0a7ca70e665f38229632aee9297ab5ab6a10f57c385: Status 404 returned error can't find the container with id 860cb70ef340ffe818faf0a7ca70e665f38229632aee9297ab5ab6a10f57c385
Sep 17 20:57:37.427499 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.427483    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:57:37.676979 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:37.676897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xdcgh" event={"ID":"84299a44-671b-4036-a730-695d26737976","Type":"ContainerStarted","Data":"860cb70ef340ffe818faf0a7ca70e665f38229632aee9297ab5ab6a10f57c385"}
Sep 17 20:57:40.696909 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:57:40.696878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-xdcgh" event={"ID":"84299a44-671b-4036-a730-695d26737976","Type":"ContainerStarted","Data":"b3d3cc0aa033e389f9298f3678b87c907161a5dce893e49d92823abed407c368"}
Sep 17 20:58:11.485289 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.485258    2582 scope.go:117] "RemoveContainer" containerID="834a23c1ada68b0fd4a9d4e606a72c7872e3b670deca095a434fe9c544f019a7"
Sep 17 20:58:11.494990 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.494962    2582 scope.go:117] "RemoveContainer" containerID="892e400c60f492c38e7584d5ac51bc46ae8ca0bad0fa7e2a0b1326847c098872"
Sep 17 20:58:11.502672 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.502648    2582 scope.go:117] "RemoveContainer" containerID="8d40636f3c99778c2c3c332a49e09acd2c3cd83249af1d17a563aa015051a903"
Sep 17 20:58:11.510244 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.510220    2582 scope.go:117] "RemoveContainer" containerID="cd1a15d10ce0c07d93981e7746d113a6734cd4ce864d05db582aa1a1e2d35053"
Sep 17 20:58:11.516711 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.516688    2582 scope.go:117] "RemoveContainer" containerID="865fe520027b4184895cd019432c42c2fe9954770d83915766ac374463000a4f"
Sep 17 20:58:11.523055 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.523040    2582 scope.go:117] "RemoveContainer" containerID="4bd382b421d000c29186ee492dea84fe12857ca3957691f5acaa98e0ca1e4dca"
Sep 17 20:58:11.529584 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:11.529568    2582 scope.go:117] "RemoveContainer" containerID="322ce91fd5612997909127c0586b1b99e7b8f749daa738cbdc5c7c7f00098c6b"
Sep 17 20:58:16.695435 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.695377    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-xdcgh" podStartSLOduration=37.561039095 podStartE2EDuration="40.695360866s" totalImagesPullingTime="3.134321771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:57:40.727227747 +0000 UTC m=+511.306463598" watchObservedRunningTime="2026-09-17 20:58:16.695360866 +0000 UTC m=+547.274596716"
Sep 17 20:58:16.695809 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.695495    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"]
Sep 17 20:58:16.700434 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.700415    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.703822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.703760    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 20:58:16.703822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.703781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:58:16.703822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.703787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 20:58:16.703822 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.703814    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 20:58:16.704089 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.703827    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 20:58:16.704158 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.704135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nkhp9\""
Sep 17 20:58:16.709496 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.709476    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"]
Sep 17 20:58:16.720079 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.720057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.720196 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.720085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.720196 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.720148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn4t\" (UniqueName: \"kubernetes.io/projected/03abbed5-ba24-41a1-99b8-849fa2d7306d-kube-api-access-bdn4t\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.720270 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.720239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03abbed5-ba24-41a1-99b8-849fa2d7306d-manager-config\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.820852 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.820813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03abbed5-ba24-41a1-99b8-849fa2d7306d-manager-config\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.820852 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.820853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.821134 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.820870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.821134 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.820898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn4t\" (UniqueName: \"kubernetes.io/projected/03abbed5-ba24-41a1-99b8-849fa2d7306d-kube-api-access-bdn4t\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.821485 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.821465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03abbed5-ba24-41a1-99b8-849fa2d7306d-manager-config\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.823361 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.823338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.823619 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.823597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/03abbed5-ba24-41a1-99b8-849fa2d7306d-metrics-cert\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:16.830290 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:16.830269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn4t\" (UniqueName: \"kubernetes.io/projected/03abbed5-ba24-41a1-99b8-849fa2d7306d-kube-api-access-bdn4t\") pod \"lws-controller-manager-7dbd69c77d-q54kg\" (UID: \"03abbed5-ba24-41a1-99b8-849fa2d7306d\") " pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:17.010152 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:17.010125    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:17.319852 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:17.319767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"]
Sep 17 20:58:17.322765 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:58:17.322738    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03abbed5_ba24_41a1_99b8_849fa2d7306d.slice/crio-e48291c28513b5be51e5bffc8a2403f66b69cd0870e5122790e1f325b01e5711 WatchSource:0}: Error finding container e48291c28513b5be51e5bffc8a2403f66b69cd0870e5122790e1f325b01e5711: Status 404 returned error can't find the container with id e48291c28513b5be51e5bffc8a2403f66b69cd0870e5122790e1f325b01e5711
Sep 17 20:58:17.851646 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:17.851613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg" event={"ID":"03abbed5-ba24-41a1-99b8-849fa2d7306d","Type":"ContainerStarted","Data":"e48291c28513b5be51e5bffc8a2403f66b69cd0870e5122790e1f325b01e5711"}
Sep 17 20:58:31.913881 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:31.913845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg" event={"ID":"03abbed5-ba24-41a1-99b8-849fa2d7306d","Type":"ContainerStarted","Data":"334cb57a47922dabb8db3260763dce3a480a0a055f518be7a8cf953ef403769e"}
Sep 17 20:58:31.914335 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:31.913945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:31.932696 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:31.932646    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg" podStartSLOduration=2.281452124 podStartE2EDuration="15.932631005s" totalImagesPullingTime="13.651178881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:31.93142906 +0000 UTC m=+562.510664910" watchObservedRunningTime="2026-09-17 20:58:31.932631005 +0000 UTC m=+562.511866856"
Sep 17 20:58:42.920363 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:42.920336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7dbd69c77d-q54kg"
Sep 17 20:58:44.699376 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.699340    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"]
Sep 17 20:58:44.702653 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.702637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.710055 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.710034    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 20:58:44.710359 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.710345    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 20:58:44.710410 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.710355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r926p\""
Sep 17 20:58:44.719348 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.719326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"]
Sep 17 20:58:44.747423 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.747393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fdf113d-4de9-4ebf-8761-e0fed7caef90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.747585 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.747429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvwd\" (UniqueName: \"kubernetes.io/projected/9fdf113d-4de9-4ebf-8761-e0fed7caef90-kube-api-access-8vvwd\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.848339 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.848291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fdf113d-4de9-4ebf-8761-e0fed7caef90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.848339 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.848342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvwd\" (UniqueName: \"kubernetes.io/projected/9fdf113d-4de9-4ebf-8761-e0fed7caef90-kube-api-access-8vvwd\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.848685 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.848667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fdf113d-4de9-4ebf-8761-e0fed7caef90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:44.856788 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:44.856762    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvwd\" (UniqueName: \"kubernetes.io/projected/9fdf113d-4de9-4ebf-8761-e0fed7caef90-kube-api-access-8vvwd\") pod \"kuadrant-operator-controller-manager-6d4cb79c-2cfxd\" (UID: \"9fdf113d-4de9-4ebf-8761-e0fed7caef90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:45.012754 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:45.012722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:45.289970 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:45.289890    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"]
Sep 17 20:58:45.293505 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:58:45.293477    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdf113d_4de9_4ebf_8761_e0fed7caef90.slice/crio-2b293b3558628abb3e2e1cb0903c90a528f573e30299007ba65f2853bd109f3d WatchSource:0}: Error finding container 2b293b3558628abb3e2e1cb0903c90a528f573e30299007ba65f2853bd109f3d: Status 404 returned error can't find the container with id 2b293b3558628abb3e2e1cb0903c90a528f573e30299007ba65f2853bd109f3d
Sep 17 20:58:45.972584 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:45.972544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd" event={"ID":"9fdf113d-4de9-4ebf-8761-e0fed7caef90","Type":"ContainerStarted","Data":"2b293b3558628abb3e2e1cb0903c90a528f573e30299007ba65f2853bd109f3d"}
Sep 17 20:58:49.998618 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:49.998577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd" event={"ID":"9fdf113d-4de9-4ebf-8761-e0fed7caef90","Type":"ContainerStarted","Data":"e9e807c2351121262bf793eb04aa1c90ef638b398a376649429f2052ebc6fe18"}
Sep 17 20:58:49.999016 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:49.998701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:58:50.020385 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:50.020337    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd" podStartSLOduration=1.7975053760000002 podStartE2EDuration="6.02031791s" totalImagesPullingTime="4.222812534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:50.018984956 +0000 UTC m=+580.598220806" watchObservedRunningTime="2026-09-17 20:58:50.02031791 +0000 UTC m=+580.599553763"
Sep 17 20:58:51.868930 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.868900    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"]
Sep 17 20:58:51.872289 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.872268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:51.874808 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.874781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 20:58:51.874808 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.874801    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 20:58:51.875018 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.874830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5r9s2\""
Sep 17 20:58:51.881797 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:51.881776    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"]
Sep 17 20:58:52.010674 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.010643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.010846 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.010692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.010846 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.010714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvfv\" (UniqueName: \"kubernetes.io/projected/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-kube-api-access-qwvfv\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.111803 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.111756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.111803 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.111807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvfv\" (UniqueName: \"kubernetes.io/projected/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-kube-api-access-qwvfv\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.112052 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.111857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.112524 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.112498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.114329 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.114305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.120958 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.120911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvfv\" (UniqueName: \"kubernetes.io/projected/0aae8fe1-d98a-471c-84c0-892b0b9b69ee-kube-api-access-qwvfv\") pod \"kuadrant-console-plugin-fbdd4f445-7hpgj\" (UID: \"0aae8fe1-d98a-471c-84c0-892b0b9b69ee\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.181764 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.181734    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"
Sep 17 20:58:52.457286 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:52.457155    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj"]
Sep 17 20:58:52.459897 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:58:52.459865    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0aae8fe1_d98a_471c_84c0_892b0b9b69ee.slice/crio-b29e179c3bfdf8e7ef38c65318026e2a1da72ecf275c289901e67ccddb711ee7 WatchSource:0}: Error finding container b29e179c3bfdf8e7ef38c65318026e2a1da72ecf275c289901e67ccddb711ee7: Status 404 returned error can't find the container with id b29e179c3bfdf8e7ef38c65318026e2a1da72ecf275c289901e67ccddb711ee7
Sep 17 20:58:53.013074 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:53.013040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj" event={"ID":"0aae8fe1-d98a-471c-84c0-892b0b9b69ee","Type":"ContainerStarted","Data":"b29e179c3bfdf8e7ef38c65318026e2a1da72ecf275c289901e67ccddb711ee7"}
Sep 17 20:58:58.037817 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:58.037775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj" event={"ID":"0aae8fe1-d98a-471c-84c0-892b0b9b69ee","Type":"ContainerStarted","Data":"807a5a589ec0d470162e06053274bace8af401cd5315442167595222b774f7de"}
Sep 17 20:58:58.058235 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:58:58.058180    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7hpgj" podStartSLOduration=1.966562373 podStartE2EDuration="7.05816369s" totalImagesPullingTime="5.091601317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:58.056792665 +0000 UTC m=+588.636028507" watchObservedRunningTime="2026-09-17 20:58:58.05816369 +0000 UTC m=+588.637399540"
Sep 17 20:59:01.004813 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:01.004782    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-2cfxd"
Sep 17 20:59:11.487487 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:11.487445    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:59:11.489075 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:11.489054    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 20:59:11.492578 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:11.492548    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:59:11.493729 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:11.493706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 20:59:37.011632 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.011555    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9bthq"]
Sep 17 20:59:37.031275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.031009    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9bthq"]
Sep 17 20:59:37.031275 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.031116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.033534 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.033512    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 20:59:37.042497 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.042472    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9bthq"]
Sep 17 20:59:37.109612 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.109577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90a417e0-8995-4958-8030-5d8c44c9fd99-config-file\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.109776 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.109631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxx6\" (UniqueName: \"kubernetes.io/projected/90a417e0-8995-4958-8030-5d8c44c9fd99-kube-api-access-6gxx6\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.210688 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.210652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90a417e0-8995-4958-8030-5d8c44c9fd99-config-file\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.210856 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.210717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxx6\" (UniqueName: \"kubernetes.io/projected/90a417e0-8995-4958-8030-5d8c44c9fd99-kube-api-access-6gxx6\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.211497 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.211468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90a417e0-8995-4958-8030-5d8c44c9fd99-config-file\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.218552 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.218527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxx6\" (UniqueName: \"kubernetes.io/projected/90a417e0-8995-4958-8030-5d8c44c9fd99-kube-api-access-6gxx6\") pod \"limitador-limitador-84994f7ffc-9bthq\" (UID: \"90a417e0-8995-4958-8030-5d8c44c9fd99\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.344470 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.344376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:37.621271 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:37.621249    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9bthq"]
Sep 17 20:59:37.624042 ip-10-0-131-59 kubenswrapper[2582]: W0917 20:59:37.624009    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a417e0_8995_4958_8030_5d8c44c9fd99.slice/crio-40e2f9d64ee7b4a6c9fdeb3d087912678edd09990f70b4893afcf635000f94a9 WatchSource:0}: Error finding container 40e2f9d64ee7b4a6c9fdeb3d087912678edd09990f70b4893afcf635000f94a9: Status 404 returned error can't find the container with id 40e2f9d64ee7b4a6c9fdeb3d087912678edd09990f70b4893afcf635000f94a9
Sep 17 20:59:38.214548 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:38.214512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq" event={"ID":"90a417e0-8995-4958-8030-5d8c44c9fd99","Type":"ContainerStarted","Data":"40e2f9d64ee7b4a6c9fdeb3d087912678edd09990f70b4893afcf635000f94a9"}
Sep 17 20:59:39.221033 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:39.221004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq" event={"ID":"90a417e0-8995-4958-8030-5d8c44c9fd99","Type":"ContainerStarted","Data":"b1b17cc6e49b4b3c8d462bd22cb206e9e31805e386f7ff28b7343c208bd6a700"}
Sep 17 20:59:39.221425 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:39.221111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 20:59:39.237825 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:39.237776    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq" podStartSLOduration=2.363486571 podStartE2EDuration="3.237760398s" totalImagesPullingTime="874.273827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:39.236776875 +0000 UTC m=+629.816012727" watchObservedRunningTime="2026-09-17 20:59:39.237760398 +0000 UTC m=+629.816996248"
Sep 17 20:59:50.226182 ip-10-0-131-59 kubenswrapper[2582]: I0917 20:59:50.226153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9bthq"
Sep 17 21:03:50.390992 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.390960    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:03:50.393848 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.393832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.397234 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.397210    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-4lmq4\""
Sep 17 21:03:50.397355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.397215    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 21:03:50.397355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.397298    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 21:03:50.397355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.397246    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 21:03:50.404616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.404597    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:03:50.527568 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.527568 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.527763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.527763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.527763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ml6z\" (UniqueName: \"kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.527763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.527698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629026 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.628996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ml6z\" (UniqueName: \"kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629390 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629516 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629574 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.629574 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.629547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.631469 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.631445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.631664 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.631644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.639696 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.639673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ml6z\" (UniqueName: \"kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z\") pod \"fc-smoke-test-kserve-7f4687897-p9dxh\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.724045 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.723978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:03:50.994214 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.994189    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:03:50.996649 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:03:50.996624    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaa7763_798a_4798_86a4_17af99ba4a05.slice/crio-2c2081e89af2fc7717d4bd29ec66f2ccacbd76535eac81456264218951a8b490 WatchSource:0}: Error finding container 2c2081e89af2fc7717d4bd29ec66f2ccacbd76535eac81456264218951a8b490: Status 404 returned error can't find the container with id 2c2081e89af2fc7717d4bd29ec66f2ccacbd76535eac81456264218951a8b490
Sep 17 21:03:50.998485 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:50.998469    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:03:51.330973 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:51.330939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" event={"ID":"dcaa7763-798a-4798-86a4-17af99ba4a05","Type":"ContainerStarted","Data":"2c2081e89af2fc7717d4bd29ec66f2ccacbd76535eac81456264218951a8b490"}
Sep 17 21:03:53.343582 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:53.343502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" event={"ID":"dcaa7763-798a-4798-86a4-17af99ba4a05","Type":"ContainerStarted","Data":"38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653"}
Sep 17 21:03:53.368159 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:53.368112    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" podStartSLOduration=1.938643329 podStartE2EDuration="3.368080188s" totalImagesPullingTime="1.429436859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:03:53.365211903 +0000 UTC m=+883.944447755" watchObservedRunningTime="2026-09-17 21:03:53.368080188 +0000 UTC m=+883.947316038"
Sep 17 21:03:53.724884 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:03:53.724796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:00.724837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:00.724746    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:00.737553 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:00.737521    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:01.389746 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:01.389718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:02.865527 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.865496    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:04:02.868668 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.868648    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:02.871753 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.871732    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 21:04:02.871928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.871802    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 21:04:02.871928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.871751    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-ndgtw\""
Sep 17 21:04:02.871928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.871753    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 21:04:02.872371 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.871743    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-mfs66\""
Sep 17 21:04:02.878662 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.878638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:04:02.926069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.926040    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrrm\" (UniqueName: \"kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:02.926226 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:02.926172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.027065 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.027028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.027222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.027080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrrm\" (UniqueName: \"kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.029471 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.029451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.035418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.035392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrrm\" (UniqueName: \"kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.180274 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.180201    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:03.458832 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:03.458783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:04:03.461195 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:04:03.461168    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2302c9_a887_47b0_b815_d26a579969d9.slice/crio-91c7fa5ca08b503ab9ff0b6399dfa18564fae65b941ae0a67c42020999508e17 WatchSource:0}: Error finding container 91c7fa5ca08b503ab9ff0b6399dfa18564fae65b941ae0a67c42020999508e17: Status 404 returned error can't find the container with id 91c7fa5ca08b503ab9ff0b6399dfa18564fae65b941ae0a67c42020999508e17
Sep 17 21:04:04.396950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:04.396912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" event={"ID":"9d2302c9-a887-47b0-b815-d26a579969d9","Type":"ContainerStarted","Data":"91c7fa5ca08b503ab9ff0b6399dfa18564fae65b941ae0a67c42020999508e17"}
Sep 17 21:04:06.408503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:06.408465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" event={"ID":"9d2302c9-a887-47b0-b815-d26a579969d9","Type":"ContainerStarted","Data":"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"}
Sep 17 21:04:06.408949 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:06.408529    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:04:06.427940 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:06.427883    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podStartSLOduration=2.044056941 podStartE2EDuration="4.427863719s" totalImagesPullingTime="2.383806778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:06.427858975 +0000 UTC m=+897.007094840" watchObservedRunningTime="2026-09-17 21:04:06.427863719 +0000 UTC m=+897.007099571"
Sep 17 21:04:11.518068 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:11.518039    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:04:11.518502 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:11.518078    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:04:11.522875 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:11.522853    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:04:11.522995 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:11.522928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:04:36.313222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.313186    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:04:36.322676 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:04:36.322642    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:04:36.322676 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:36.322676 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:36.322676 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653" cmd=["/bin/sleep","15"]
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.322698    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" message="" err=<
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:04:36.322710    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main" containerID="cri-o://38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653"
Sep 17 21:04:36.322920 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.322774    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main" containerID="cri-o://38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653" gracePeriod=60
Sep 17 21:04:36.378798 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.378753    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Sep 17 21:04:36.541622 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.541597    2582 generic.go:356] "Generic (PLEG): container finished" podID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerID="38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653" exitCode=0
Sep 17 21:04:36.541732 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.541665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" event={"ID":"dcaa7763-798a-4798-86a4-17af99ba4a05","Type":"ContainerDied","Data":"38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653"}
Sep 17 21:04:36.672357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.672337    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:36.813244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813210    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813250    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ml6z\" (UniqueName: \"kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813361    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home\") pod \"dcaa7763-798a-4798-86a4-17af99ba4a05\" (UID: \"dcaa7763-798a-4798-86a4-17af99ba4a05\") "
Sep 17 21:04:36.813673 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:36.813736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813670    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:36.813736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home" (OuterVolumeSpecName: "home") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:36.813736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.813720    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:36.815598 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.815578    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:36.815865 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.815841    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z" (OuterVolumeSpecName: "kube-api-access-8ml6z") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "kube-api-access-8ml6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:36.815924 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.815848    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm" (OuterVolumeSpecName: "dshm") pod "dcaa7763-798a-4798-86a4-17af99ba4a05" (UID: "dcaa7763-798a-4798-86a4-17af99ba4a05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:36.914901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.914824    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:36.914901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.914854    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ml6z\" (UniqueName: \"kubernetes.io/projected/dcaa7763-798a-4798-86a4-17af99ba4a05-kube-api-access-8ml6z\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:36.914901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.914866    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcaa7763-798a-4798-86a4-17af99ba4a05-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:36.914901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.914874    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:36.914901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:36.914882    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcaa7763-798a-4798-86a4-17af99ba4a05-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:04:37.413819 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.413781    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:04:37.548465 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.548436    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"
Sep 17 21:04:37.548613 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.548474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh" event={"ID":"dcaa7763-798a-4798-86a4-17af99ba4a05","Type":"ContainerDied","Data":"2c2081e89af2fc7717d4bd29ec66f2ccacbd76535eac81456264218951a8b490"}
Sep 17 21:04:37.548613 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.548533    2582 scope.go:117] "RemoveContainer" containerID="38f3796275ffc31d016d0225697925162c3125a2f59a1d16112defcd3fd48653"
Sep 17 21:04:37.569731 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.569710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:04:37.576080 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.576057    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-p9dxh"]
Sep 17 21:04:37.584051 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:37.584030    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" path="/var/lib/kubelet/pods/dcaa7763-798a-4798-86a4-17af99ba4a05/volumes"
Sep 17 21:04:38.417151 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:38.417120    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:04:38.418906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:38.418876    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:04:48.419429 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:48.419393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:04:53.436457 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:53.436395    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" containerID="cri-o://8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0" gracePeriod=45
Sep 17 21:04:58.419283 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:04:58.419242    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:08.419195 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:08.419156    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:18.418981 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:18.418948    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:05:23.680329 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.680306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:05:23.746341 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.746272    2582 generic.go:356] "Generic (PLEG): container finished" podID="9d2302c9-a887-47b0-b815-d26a579969d9" containerID="8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0" exitCode=0
Sep 17 21:05:23.746341 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.746309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"
Sep 17 21:05:23.746506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.746344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" event={"ID":"9d2302c9-a887-47b0-b815-d26a579969d9","Type":"ContainerDied","Data":"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"}
Sep 17 21:05:23.746506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.746395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5" event={"ID":"9d2302c9-a887-47b0-b815-d26a579969d9","Type":"ContainerDied","Data":"91c7fa5ca08b503ab9ff0b6399dfa18564fae65b941ae0a67c42020999508e17"}
Sep 17 21:05:23.746506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.746440    2582 scope.go:117] "RemoveContainer" containerID="8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"
Sep 17 21:05:23.754591 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.754572    2582 scope.go:117] "RemoveContainer" containerID="8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"
Sep 17 21:05:23.754872 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:05:23.754852    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0\": container with ID starting with 8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0 not found: ID does not exist" containerID="8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"
Sep 17 21:05:23.754965 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.754879    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0"} err="failed to get container status \"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0\": rpc error: code = NotFound desc = could not find container \"8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0\": container with ID starting with 8040672ac21bdcba0064b30eb624299b0332f25cbaacfb707963d07c5ffd78d0 not found: ID does not exist"
Sep 17 21:05:23.786723 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.786703    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdrrm\" (UniqueName: \"kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm\") pod \"9d2302c9-a887-47b0-b815-d26a579969d9\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") "
Sep 17 21:05:23.786856 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.786777    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs\") pod \"9d2302c9-a887-47b0-b815-d26a579969d9\" (UID: \"9d2302c9-a887-47b0-b815-d26a579969d9\") "
Sep 17 21:05:23.788859 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.788830    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d2302c9-a887-47b0-b815-d26a579969d9" (UID: "9d2302c9-a887-47b0-b815-d26a579969d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:05:23.788950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.788839    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm" (OuterVolumeSpecName: "kube-api-access-cdrrm") pod "9d2302c9-a887-47b0-b815-d26a579969d9" (UID: "9d2302c9-a887-47b0-b815-d26a579969d9"). InnerVolumeSpecName "kube-api-access-cdrrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:05:23.888105 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.888054    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2302c9-a887-47b0-b815-d26a579969d9-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:05:23.888105 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:23.888090    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrrm\" (UniqueName: \"kubernetes.io/projected/9d2302c9-a887-47b0-b815-d26a579969d9-kube-api-access-cdrrm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:05:24.065998 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:24.065967    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:05:24.069565 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:24.069541    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5rwdv5"]
Sep 17 21:05:25.584048 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:25.584013    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" path="/var/lib/kubelet/pods/9d2302c9-a887-47b0-b815-d26a579969d9/volumes"
Sep 17 21:05:54.394623 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.394590    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.394929    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main"
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.394941    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main"
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.394960    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main"
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.394966    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main"
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.395013    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaa7763-798a-4798-86a4-17af99ba4a05" containerName="main"
Sep 17 21:05:54.395066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.395023    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2302c9-a887-47b0-b815-d26a579969d9" containerName="main"
Sep 17 21:05:54.397946 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.397931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.401144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.401115    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-5cr8t\""
Sep 17 21:05:54.401144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.401135    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 21:05:54.401144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.401142    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 21:05:54.401355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.401142    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 21:05:54.406986 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.406963    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:05:54.530488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hbx\" (UniqueName: \"kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.530652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.530652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.530652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.530652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.530652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.530627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hbx\" (UniqueName: \"kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631722 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631774 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.631810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.631764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.633610 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.633591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.633863 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.633846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.639213 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.639195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hbx\" (UniqueName: \"kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx\") pod \"fc-concurrency-test-kserve-b5d44467c-n97ml\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.709518 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.709432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:05:54.979854 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:54.979830    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:05:54.982474 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:05:54.982445    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51621fcf_aee6_4f0e_8759_1c947130f6af.slice/crio-34f9525916c9bb597736e1626c25187570679689e636816b976ac7de3db48398 WatchSource:0}: Error finding container 34f9525916c9bb597736e1626c25187570679689e636816b976ac7de3db48398: Status 404 returned error can't find the container with id 34f9525916c9bb597736e1626c25187570679689e636816b976ac7de3db48398
Sep 17 21:05:55.893441 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:55.893406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" event={"ID":"51621fcf-aee6-4f0e-8759-1c947130f6af","Type":"ContainerStarted","Data":"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"}
Sep 17 21:05:55.893910 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:55.893468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" event={"ID":"51621fcf-aee6-4f0e-8759-1c947130f6af","Type":"ContainerStarted","Data":"34f9525916c9bb597736e1626c25187570679689e636816b976ac7de3db48398"}
Sep 17 21:05:55.916487 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:55.916427    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" podStartSLOduration=1.916410021 podStartE2EDuration="1.916410021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:05:55.915350144 +0000 UTC m=+1006.494585998" watchObservedRunningTime="2026-09-17 21:05:55.916410021 +0000 UTC m=+1006.495645869"
Sep 17 21:05:56.709957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:05:56.709922    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:04.709878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:04.709833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:04.722597 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:04.722564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:04.947782 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:04.947747    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:57.829072 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:57.828991    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:06:57.838048 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:06:57.838014    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:06:57.838048 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:06:57.838048 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:06:57.838048 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6" cmd=["/bin/sleep","15"]
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:57.838067    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" message="" err=<
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:06:57.838080    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main" containerID="cri-o://0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"
Sep 17 21:06:57.838427 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:57.838134    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main" containerID="cri-o://0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6" gracePeriod=60
Sep 17 21:06:57.937228 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:57.937193    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main" probeResult="failure" output="Get \"https://10.133.0.28:8000/health\": dial tcp 10.133.0.28:8000: connect: connection refused"
Sep 17 21:06:58.178447 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.178419    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:58.179588 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.179565    2582 generic.go:356] "Generic (PLEG): container finished" podID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerID="0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6" exitCode=0
Sep 17 21:06:58.179695 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.179639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" event={"ID":"51621fcf-aee6-4f0e-8759-1c947130f6af","Type":"ContainerDied","Data":"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"}
Sep 17 21:06:58.179695 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.179688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml" event={"ID":"51621fcf-aee6-4f0e-8759-1c947130f6af","Type":"ContainerDied","Data":"34f9525916c9bb597736e1626c25187570679689e636816b976ac7de3db48398"}
Sep 17 21:06:58.179771 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.179716    2582 scope.go:117] "RemoveContainer" containerID="0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"
Sep 17 21:06:58.187766 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.187748    2582 scope.go:117] "RemoveContainer" containerID="0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"
Sep 17 21:06:58.188043 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:06:58.188022    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6\": container with ID starting with 0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6 not found: ID does not exist" containerID="0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"
Sep 17 21:06:58.188089 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.188052    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6"} err="failed to get container status \"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6\": rpc error: code = NotFound desc = could not find container \"0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6\": container with ID starting with 0395dde431f9b9114b8e60233e2a55e0306efb11e0ff0cdc29ec183504ad00a6 not found: ID does not exist"
Sep 17 21:06:58.343351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8hbx\" (UniqueName: \"kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343357    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343425    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache\") pod \"51621fcf-aee6-4f0e-8759-1c947130f6af\" (UID: \"51621fcf-aee6-4f0e-8759-1c947130f6af\") "
Sep 17 21:06:58.343805 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:58.343865 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343804    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:58.343979 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.343942    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache" (OuterVolumeSpecName: "model-cache") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:58.344056 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.344034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home" (OuterVolumeSpecName: "home") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:58.345764 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.345732    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx" (OuterVolumeSpecName: "kube-api-access-p8hbx") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "kube-api-access-p8hbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:58.345881 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.345764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm" (OuterVolumeSpecName: "dshm") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:58.346050 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.346032    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51621fcf-aee6-4f0e-8759-1c947130f6af" (UID: "51621fcf-aee6-4f0e-8759-1c947130f6af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:58.445002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.444971    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51621fcf-aee6-4f0e-8759-1c947130f6af-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:58.445002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.445000    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:58.445002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.445009    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hbx\" (UniqueName: \"kubernetes.io/projected/51621fcf-aee6-4f0e-8759-1c947130f6af-kube-api-access-p8hbx\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:58.445252 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.445020    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:58.445252 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:58.445028    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51621fcf-aee6-4f0e-8759-1c947130f6af-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:06:59.184331 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:59.184303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"
Sep 17 21:06:59.204439 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:59.204414    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:06:59.210111 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:59.210077    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-n97ml"]
Sep 17 21:06:59.584707 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:06:59.584675    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" path="/var/lib/kubelet/pods/51621fcf-aee6-4f0e-8759-1c947130f6af/volumes"
Sep 17 21:08:55.749940 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.749906    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:08:55.750528 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.750429    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main"
Sep 17 21:08:55.750528 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.750449    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main"
Sep 17 21:08:55.750654 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.750538    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="51621fcf-aee6-4f0e-8759-1c947130f6af" containerName="main"
Sep 17 21:08:55.753431 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.753411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.756798 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.756777    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 21:08:55.756908 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.756823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 21:08:55.756908 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.756858    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 21:08:55.756908 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.756783    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-j6wnd\""
Sep 17 21:08:55.762869 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.762849    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:08:55.904756 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.904916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.904916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.904916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572sb\" (UniqueName: \"kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.904916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:55.905065 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:55.904966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.005915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.005915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572sb\" (UniqueName: \"kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.005915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.005915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.006262 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.006262 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.005988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.006362 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.006344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.006424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.006408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.006525 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.006506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.008252 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.008231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.008519 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.008504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.013370 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.013350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572sb\" (UniqueName: \"kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.064423 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.064388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:08:56.339836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.339765    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:08:56.343129 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:08:56.343080    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c81a17_205c_4bf7_a9d4_f64e5e9534ea.slice/crio-322c45904d8fb51a1a0b33366ab20d34355eb7196a865a2f3387b35ba5c7c1c7 WatchSource:0}: Error finding container 322c45904d8fb51a1a0b33366ab20d34355eb7196a865a2f3387b35ba5c7c1c7: Status 404 returned error can't find the container with id 322c45904d8fb51a1a0b33366ab20d34355eb7196a865a2f3387b35ba5c7c1c7
Sep 17 21:08:56.710479 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.710392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" event={"ID":"55c81a17-205c-4bf7-a9d4-f64e5e9534ea","Type":"ContainerStarted","Data":"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"}
Sep 17 21:08:56.710479 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.710467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" event={"ID":"55c81a17-205c-4bf7-a9d4-f64e5e9534ea","Type":"ContainerStarted","Data":"322c45904d8fb51a1a0b33366ab20d34355eb7196a865a2f3387b35ba5c7c1c7"}
Sep 17 21:08:56.729543 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:56.729498    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" podStartSLOduration=1.729483749 podStartE2EDuration="1.729483749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:56.728205319 +0000 UTC m=+1187.307441170" watchObservedRunningTime="2026-09-17 21:08:56.729483749 +0000 UTC m=+1187.308719600"
Sep 17 21:08:57.065155 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:08:57.065121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:09:06.064982 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:06.064947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:09:06.077733 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:06.077710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:09:06.770484 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:06.770457    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:09:11.545462 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:11.545424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:09:11.546526 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:11.546503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:09:11.550139 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:11.550119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:09:11.551175 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:09:11.551153    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:10:56.493593 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:56.493558    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:10:56.502454 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:10:56.502414    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:10:56.502454 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.502454 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.502454 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868" cmd=["/bin/sleep","15"]
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:56.502472    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" message="" err=<
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:10:56.502484    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main" containerID="cri-o://53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"
Sep 17 21:10:56.502727 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:56.502531    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main" containerID="cri-o://53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868" gracePeriod=60
Sep 17 21:10:56.759678 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:56.759643    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Sep 17 21:10:56.856286 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:56.856260    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:10:57.007353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007515 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007515 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007515 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007457    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007515 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007731 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-572sb\" (UniqueName: \"kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb\") pod \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\" (UID: \"55c81a17-205c-4bf7-a9d4-f64e5e9534ea\") "
Sep 17 21:10:57.007731 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home" (OuterVolumeSpecName: "home") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.007836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007731    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.007836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.007836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007808    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.007836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.007825    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.009803 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.009738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:57.009803 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.009787    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb" (OuterVolumeSpecName: "kube-api-access-572sb") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "kube-api-access-572sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:10:57.009940 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.009803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm" (OuterVolumeSpecName: "dshm") pod "55c81a17-205c-4bf7-a9d4-f64e5e9534ea" (UID: "55c81a17-205c-4bf7-a9d4-f64e5e9534ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:57.108843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.108803    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.108843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.108840    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572sb\" (UniqueName: \"kubernetes.io/projected/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-kube-api-access-572sb\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.108843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.108850    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.108843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.108859    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c81a17-205c-4bf7-a9d4-f64e5e9534ea-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:10:57.254605 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.254564    2582 generic.go:356] "Generic (PLEG): container finished" podID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerID="53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868" exitCode=0
Sep 17 21:10:57.254752 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.254609    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"
Sep 17 21:10:57.254752 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.254623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" event={"ID":"55c81a17-205c-4bf7-a9d4-f64e5e9534ea","Type":"ContainerDied","Data":"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"}
Sep 17 21:10:57.254752 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.254674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj" event={"ID":"55c81a17-205c-4bf7-a9d4-f64e5e9534ea","Type":"ContainerDied","Data":"322c45904d8fb51a1a0b33366ab20d34355eb7196a865a2f3387b35ba5c7c1c7"}
Sep 17 21:10:57.254752 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.254697    2582 scope.go:117] "RemoveContainer" containerID="53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"
Sep 17 21:10:57.263835 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.263815    2582 scope.go:117] "RemoveContainer" containerID="53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"
Sep 17 21:10:57.264143 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:10:57.264125    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868\": container with ID starting with 53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868 not found: ID does not exist" containerID="53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"
Sep 17 21:10:57.264204 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.264151    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868"} err="failed to get container status \"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868\": rpc error: code = NotFound desc = could not find container \"53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868\": container with ID starting with 53d80ef4528c3f299508b0a2de34073ae71d082bb56ca8735f7e0406bc958868 not found: ID does not exist"
Sep 17 21:10:57.276945 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.276923    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:10:57.282850 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.282831    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-4qhcj"]
Sep 17 21:10:57.583789 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:10:57.583714    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" path="/var/lib/kubelet/pods/55c81a17-205c-4bf7-a9d4-f64e5e9534ea/volumes"
Sep 17 21:12:05.548394 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.548362    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:05.548914 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.548681    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main"
Sep 17 21:12:05.548914 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.548692    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main"
Sep 17 21:12:05.548914 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.548760    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c81a17-205c-4bf7-a9d4-f64e5e9534ea" containerName="main"
Sep 17 21:12:05.552017 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.551993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.555499 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.555129    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 21:12:05.555499 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.555197    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 21:12:05.555499 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.555373    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-m5tsj\""
Sep 17 21:12:05.555881 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.555863    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 21:12:05.555948 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.555879    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-l6fbg\""
Sep 17 21:12:05.563076 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.563054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:05.664873 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.664840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6bm\" (UniqueName: \"kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.665049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.664914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.665049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.664942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.665049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.664967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.665049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.664989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.665049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.665020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766017 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.765988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6bm\" (UniqueName: \"kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766550 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766649 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.766714 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.766676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.768992 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.768957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.769172 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.769155    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.777962 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.777937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6bm\" (UniqueName: \"kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:05.862711 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:05.862634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:06.133601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:06.133578    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:06.136542 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:12:06.136516    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8c11a3_5a27_43cb_be88_eaa054500793.slice/crio-3dbcf4e4e3717a0c1202356546b6d8273e7983d3b9bda6f18d962d04f3e83f09 WatchSource:0}: Error finding container 3dbcf4e4e3717a0c1202356546b6d8273e7983d3b9bda6f18d962d04f3e83f09: Status 404 returned error can't find the container with id 3dbcf4e4e3717a0c1202356546b6d8273e7983d3b9bda6f18d962d04f3e83f09
Sep 17 21:12:06.139064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:06.139046    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:12:06.563130 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:06.563073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerStarted","Data":"3dbcf4e4e3717a0c1202356546b6d8273e7983d3b9bda6f18d962d04f3e83f09"}
Sep 17 21:12:08.574035 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:08.573997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerStarted","Data":"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"}
Sep 17 21:12:08.574529 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:08.574112    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:09.584411 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:09.584378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerStarted","Data":"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"}
Sep 17 21:12:09.600049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:09.599984    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" podStartSLOduration=3.207459766 podStartE2EDuration="4.59996715s" totalImagesPullingTime="1.392507384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:09.598633118 +0000 UTC m=+1380.177868970" watchObservedRunningTime="2026-09-17 21:12:09.59996715 +0000 UTC m=+1380.179203038"
Sep 17 21:12:09.863518 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:09.863431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:15.863268 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:15.863187    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:15.875465 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:15.875440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:16.620571 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:16.620545    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:17.625535 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:17.625504    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:53.261350 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.261265    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:53.271916 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:12:53.271876    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:12:53.271916 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.271916 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.271916 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354" cmd=["/bin/sleep","15"]
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.271939    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" message="" err=<
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:12:53.271955    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main" containerID="cri-o://7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"
Sep 17 21:12:53.272165 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.272013    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main" containerID="cri-o://7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354" gracePeriod=60
Sep 17 21:12:53.483506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.483447    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="llm-d-routing-sidecar" containerID="cri-o://860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086" gracePeriod=60
Sep 17 21:12:53.610416 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.610378    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8001/health\": dial tcp 10.133.0.30:8001: connect: connection refused"
Sep 17 21:12:53.725087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.725063    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:53.775641 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775608    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354" exitCode=0
Sep 17 21:12:53.775641 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775638    2582 generic.go:356] "Generic (PLEG): container finished" podID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerID="860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086" exitCode=0
Sep 17 21:12:53.775838 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775658    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"
Sep 17 21:12:53.775838 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerDied","Data":"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"}
Sep 17 21:12:53.775838 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerDied","Data":"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"}
Sep 17 21:12:53.775838 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg" event={"ID":"4f8c11a3-5a27-43cb-be88-eaa054500793","Type":"ContainerDied","Data":"3dbcf4e4e3717a0c1202356546b6d8273e7983d3b9bda6f18d962d04f3e83f09"}
Sep 17 21:12:53.775838 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.775806    2582 scope.go:117] "RemoveContainer" containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"
Sep 17 21:12:53.785313 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.785299    2582 scope.go:117] "RemoveContainer" containerID="860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"
Sep 17 21:12:53.793006 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.792992    2582 scope.go:117] "RemoveContainer" containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"
Sep 17 21:12:53.793360 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:12:53.793342    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354\": container with ID starting with 7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354 not found: ID does not exist" containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"
Sep 17 21:12:53.793434 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793367    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"} err="failed to get container status \"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354\": rpc error: code = NotFound desc = could not find container \"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354\": container with ID starting with 7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354 not found: ID does not exist"
Sep 17 21:12:53.793434 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793385    2582 scope.go:117] "RemoveContainer" containerID="860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"
Sep 17 21:12:53.793662 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:12:53.793643    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086\": container with ID starting with 860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086 not found: ID does not exist" containerID="860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"
Sep 17 21:12:53.793721 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793670    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"} err="failed to get container status \"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086\": rpc error: code = NotFound desc = could not find container \"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086\": container with ID starting with 860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086 not found: ID does not exist"
Sep 17 21:12:53.793721 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793690    2582 scope.go:117] "RemoveContainer" containerID="7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"
Sep 17 21:12:53.793950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793929    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354"} err="failed to get container status \"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354\": rpc error: code = NotFound desc = could not find container \"7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354\": container with ID starting with 7c379a78bb427f017e8cf4dd0cfde1bce9e29172a1eb57d17d771325aa117354 not found: ID does not exist"
Sep 17 21:12:53.794034 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.793953    2582 scope.go:117] "RemoveContainer" containerID="860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"
Sep 17 21:12:53.794307 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.794285    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086"} err="failed to get container status \"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086\": rpc error: code = NotFound desc = could not find container \"860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086\": container with ID starting with 860009559bf450ae12f0a3171a8af8610c744b1483be9155d43af02cfaf3b086 not found: ID does not exist"
Sep 17 21:12:53.850694 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850614    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm6bm\" (UniqueName: \"kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.850694 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.850694 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.850947 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850732    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.850947 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.850947 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.850802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home\") pod \"4f8c11a3-5a27-43cb-be88-eaa054500793\" (UID: \"4f8c11a3-5a27-43cb-be88-eaa054500793\") "
Sep 17 21:12:53.851139 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.851018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.851139 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.851061    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.851218 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.851161    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home" (OuterVolumeSpecName: "home") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.853044 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.853019    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm" (OuterVolumeSpecName: "dshm") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:12:53.853155 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.853055    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm" (OuterVolumeSpecName: "kube-api-access-lm6bm") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "kube-api-access-lm6bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:12:53.853155 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.853061    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f8c11a3-5a27-43cb-be88-eaa054500793" (UID: "4f8c11a3-5a27-43cb-be88-eaa054500793"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:12:53.952422 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952390    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.952422 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952415    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.952422 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952425    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f8c11a3-5a27-43cb-be88-eaa054500793-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.952422 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952432    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.952667 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952440    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f8c11a3-5a27-43cb-be88-eaa054500793-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:53.952667 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:53.952448    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm6bm\" (UniqueName: \"kubernetes.io/projected/4f8c11a3-5a27-43cb-be88-eaa054500793-kube-api-access-lm6bm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:12:54.096612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:54.096584    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:54.100790 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:54.100735    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zjxgg"]
Sep 17 21:12:55.583990 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:12:55.583950    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" path="/var/lib/kubelet/pods/4f8c11a3-5a27-43cb-be88-eaa054500793/volumes"
Sep 17 21:14:11.573304 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:11.573274    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:14:11.575118 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:11.575082    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:14:11.578304 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:11.578284    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:14:11.580451 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:11.580430    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:14:12.371547 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.371519    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:14:12.371917 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.371901    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main"
Sep 17 21:14:12.372002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.371920    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main"
Sep 17 21:14:12.372002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.371940    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="llm-d-routing-sidecar"
Sep 17 21:14:12.372002 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.371948    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="llm-d-routing-sidecar"
Sep 17 21:14:12.372183 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.372037    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="llm-d-routing-sidecar"
Sep 17 21:14:12.372183 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.372055    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8c11a3-5a27-43cb-be88-eaa054500793" containerName="main"
Sep 17 21:14:12.375174 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.375153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.378332 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.378309    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 21:14:12.378456 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.378309    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qm6fb\""
Sep 17 21:14:12.378456 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.378314    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 21:14:12.378456 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.378352    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:14:12.385364 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.385340    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:14:12.511021 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.510988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.511021 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.511026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.511253 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.511087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.511253 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.511186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612001 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.611948    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612398 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612398 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612398 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612505 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612505 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612473    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.612720 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.612704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.614543 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.614525    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.687306 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.687210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:14:12.962830 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:12.962805    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:14:12.965293 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:14:12.965264    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12b5666_8efa_4c91_8321_1aead5d5e672.slice/crio-c09baf426fb819fb0ba902d5502fdeb864dc082b74cad8923e8439e16ca5fba6 WatchSource:0}: Error finding container c09baf426fb819fb0ba902d5502fdeb864dc082b74cad8923e8439e16ca5fba6: Status 404 returned error can't find the container with id c09baf426fb819fb0ba902d5502fdeb864dc082b74cad8923e8439e16ca5fba6
Sep 17 21:14:13.132637 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:13.132597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerStarted","Data":"c09baf426fb819fb0ba902d5502fdeb864dc082b74cad8923e8439e16ca5fba6"}
Sep 17 21:14:17.160547 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:17.160504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerStarted","Data":"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"}
Sep 17 21:14:19.172628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:19.172595    2582 generic.go:356] "Generic (PLEG): container finished" podID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerID="746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2" exitCode=0
Sep 17 21:14:19.172989 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:14:19.172672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerDied","Data":"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"}
Sep 17 21:15:07.434001 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:07.433967    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerStarted","Data":"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"}
Sep 17 21:15:07.434471 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:07.434197    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:15:38.449039 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:38.449009    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:15:38.471025 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:38.470971    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" podStartSLOduration=33.297244872 podStartE2EDuration="1m26.470955173s" totalImagesPullingTime="51.088124802s" totalInitContainerRuntime="2.085585499s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:14:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:15:07.454694089 +0000 UTC m=+1558.033930000" watchObservedRunningTime="2026-09-17 21:15:38.470955173 +0000 UTC m=+1589.050191024"
Sep 17 21:15:39.935663 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:39.935639    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75_d12b5666-8efa-4c91-8321-1aead5d5e672/main/0.log"
Sep 17 21:15:40.153934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.153902    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:15:40.154202 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.154172    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="main" containerID="cri-o://13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4" gracePeriod=30
Sep 17 21:15:40.795070 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.795047    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:15:40.888762 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.888691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir\") pod \"d12b5666-8efa-4c91-8321-1aead5d5e672\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") "
Sep 17 21:15:40.888762 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.888722    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert\") pod \"d12b5666-8efa-4c91-8321-1aead5d5e672\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") "
Sep 17 21:15:40.888762 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.888747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs\") pod \"d12b5666-8efa-4c91-8321-1aead5d5e672\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") "
Sep 17 21:15:40.888999 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.888791    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location\") pod \"d12b5666-8efa-4c91-8321-1aead5d5e672\" (UID: \"d12b5666-8efa-4c91-8321-1aead5d5e672\") "
Sep 17 21:15:40.889078 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.889055    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d12b5666-8efa-4c91-8321-1aead5d5e672" (UID: "d12b5666-8efa-4c91-8321-1aead5d5e672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.889221 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.889200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d12b5666-8efa-4c91-8321-1aead5d5e672" (UID: "d12b5666-8efa-4c91-8321-1aead5d5e672"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:15:40.890969 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.890951    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d12b5666-8efa-4c91-8321-1aead5d5e672" (UID: "d12b5666-8efa-4c91-8321-1aead5d5e672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:15:40.909729 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.909700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12b5666-8efa-4c91-8321-1aead5d5e672" (UID: "d12b5666-8efa-4c91-8321-1aead5d5e672"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:15:40.990144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.990116    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.990144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.990140    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12b5666-8efa-4c91-8321-1aead5d5e672-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.990144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.990149    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12b5666-8efa-4c91-8321-1aead5d5e672-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:15:40.990499 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:40.990160    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12b5666-8efa-4c91-8321-1aead5d5e672-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:15:41.596773 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.596745    2582 generic.go:356] "Generic (PLEG): container finished" podID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerID="13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4" exitCode=0
Sep 17 21:15:41.596894 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.596792    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"
Sep 17 21:15:41.596894 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.596808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerDied","Data":"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"}
Sep 17 21:15:41.596894 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.596846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75" event={"ID":"d12b5666-8efa-4c91-8321-1aead5d5e672","Type":"ContainerDied","Data":"c09baf426fb819fb0ba902d5502fdeb864dc082b74cad8923e8439e16ca5fba6"}
Sep 17 21:15:41.596894 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.596867    2582 scope.go:117] "RemoveContainer" containerID="13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"
Sep 17 21:15:41.605481 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.605465    2582 scope.go:117] "RemoveContainer" containerID="746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"
Sep 17 21:15:41.613487 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.613456    2582 scope.go:117] "RemoveContainer" containerID="13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"
Sep 17 21:15:41.613554 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.613530    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:15:41.613720 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:15:41.613703    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4\": container with ID starting with 13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4 not found: ID does not exist" containerID="13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"
Sep 17 21:15:41.613766 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.613731    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4"} err="failed to get container status \"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4\": rpc error: code = NotFound desc = could not find container \"13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4\": container with ID starting with 13bad947b819bd4217f4db29b08d8576646de20e511252d2a97e3308f4edd3b4 not found: ID does not exist"
Sep 17 21:15:41.613766 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.613750    2582 scope.go:117] "RemoveContainer" containerID="746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"
Sep 17 21:15:41.613989 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:15:41.613971    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2\": container with ID starting with 746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2 not found: ID does not exist" containerID="746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"
Sep 17 21:15:41.614029 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.613994    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2"} err="failed to get container status \"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2\": rpc error: code = NotFound desc = could not find container \"746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2\": container with ID starting with 746ceeb03da161982bff4b4b6fcd012df37d47128eff7fc7e465c48ea7d405b2 not found: ID does not exist"
Sep 17 21:15:41.620049 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:41.620029    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7b87675dcb-hqw75"]
Sep 17 21:15:43.589234 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:15:43.589202    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" path="/var/lib/kubelet/pods/d12b5666-8efa-4c91-8321-1aead5d5e672/volumes"
Sep 17 21:16:37.132336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132305    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:16:37.132797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132635    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="main"
Sep 17 21:16:37.132797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="main"
Sep 17 21:16:37.132797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132656    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="storage-initializer"
Sep 17 21:16:37.132797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132663    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="storage-initializer"
Sep 17 21:16:37.132797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.132714    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12b5666-8efa-4c91-8321-1aead5d5e672" containerName="main"
Sep 17 21:16:37.135023 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.135006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.138533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.138513    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 21:16:37.139356 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.139337    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 21:16:37.139609 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.139594    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:16:37.139655 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.139607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 21:16:37.140328 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.140311    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 21:16:37.140431 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.140373    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-bh28s\""
Sep 17 21:16:37.150504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150610 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150610 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fpk\" (UniqueName: \"kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150610 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150743 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150743 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150743 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.150878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.150871    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:16:37.251561 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fpk\" (UniqueName: \"kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.251709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.251710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.252080 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.252020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.252164 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.252082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.252223 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.252192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.252223 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.252205    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.252470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.252447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.254121 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.254088    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.254304 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.254287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.264364 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.264343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fpk\" (UniqueName: \"kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk\") pod \"tokenizer-migration-path-test-kserve-7f478cf788-4rsc6\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.393601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.393527    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:16:37.396503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.396482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.415932 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.415909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:16:37.444711 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.444683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:37.453755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.453730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.453871 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.453784    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.453921 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.453866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.454006 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.453987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555534 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555534 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555534 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.555935 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.555886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.556078 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.556062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.557826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.557809    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.707334 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.707267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:37.719413 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.719391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:16:37.721165 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:16:37.721141    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49969ad_c510_401b_ba11_29b2a26cfb4f.slice/crio-672953be58df85ef6800d977254a1fa0e8a98209ef591f0143892841c960dc8d WatchSource:0}: Error finding container 672953be58df85ef6800d977254a1fa0e8a98209ef591f0143892841c960dc8d: Status 404 returned error can't find the container with id 672953be58df85ef6800d977254a1fa0e8a98209ef591f0143892841c960dc8d
Sep 17 21:16:37.857295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.857257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerStarted","Data":"201b1e900a4678a134d11691c0c44e93ff512df944c976a9b917aa5391b3b17b"}
Sep 17 21:16:37.857453 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.857336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerStarted","Data":"672953be58df85ef6800d977254a1fa0e8a98209ef591f0143892841c960dc8d"}
Sep 17 21:16:37.983379 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:37.983303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:16:37.987434 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:16:37.987402    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c01db3_994d_43d3_b179_1bf4c3d2d404.slice/crio-afcc15ad7999e3393c4a127460039fbfbff31e3e9490cb5d1e596e1e29918e2d WatchSource:0}: Error finding container afcc15ad7999e3393c4a127460039fbfbff31e3e9490cb5d1e596e1e29918e2d: Status 404 returned error can't find the container with id afcc15ad7999e3393c4a127460039fbfbff31e3e9490cb5d1e596e1e29918e2d
Sep 17 21:16:38.865210 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:38.865167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerStarted","Data":"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"}
Sep 17 21:16:38.865715 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:38.865240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerStarted","Data":"afcc15ad7999e3393c4a127460039fbfbff31e3e9490cb5d1e596e1e29918e2d"}
Sep 17 21:16:40.878435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:40.878403    2582 generic.go:356] "Generic (PLEG): container finished" podID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerID="201b1e900a4678a134d11691c0c44e93ff512df944c976a9b917aa5391b3b17b" exitCode=0
Sep 17 21:16:40.878850 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:40.878480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerDied","Data":"201b1e900a4678a134d11691c0c44e93ff512df944c976a9b917aa5391b3b17b"}
Sep 17 21:16:40.880036 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:40.880017    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerID="9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df" exitCode=0
Sep 17 21:16:40.880158 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:40.880108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerDied","Data":"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"}
Sep 17 21:16:41.886356 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:41.886317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerStarted","Data":"bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa"}
Sep 17 21:16:41.888281 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:41.888254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerStarted","Data":"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"}
Sep 17 21:16:41.888485 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:41.888467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:16:41.908529 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:41.908474    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" podStartSLOduration=2.502926497 podStartE2EDuration="4.908455394s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.405528897s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:41.906756882 +0000 UTC m=+1652.485992735" watchObservedRunningTime="2026-09-17 21:16:41.908455394 +0000 UTC m=+1652.487691249"
Sep 17 21:16:41.924669 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:41.924629    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" podStartSLOduration=2.548533549 podStartE2EDuration="4.92461651s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.376082961s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:41.923448601 +0000 UTC m=+1652.502684454" watchObservedRunningTime="2026-09-17 21:16:41.92461651 +0000 UTC m=+1652.503852361"
Sep 17 21:16:42.445410 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:42.445377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:47.445810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:47.445770    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:47.458676 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:47.458651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:16:47.929964 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:16:47.929938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:17:12.903842 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:12.903809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:17:14.238858 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.238825    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865_a5c01db3-994d-43d3-b179-1bf4c3d2d404/main/0.log"
Sep 17 21:17:14.796671 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.796640    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:17:14.805295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.805265    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:17:14.805483 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.805446    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="main" containerID="cri-o://8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d" gracePeriod=30
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:17:14.805761    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa" cmd=["/bin/sleep","15"]
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.805812    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" message="" err=<
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:17:14.805824    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main" containerID="cri-o://bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa"
Sep 17 21:17:14.806312 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.805857    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main" containerID="cri-o://bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa" gracePeriod=60
Sep 17 21:17:14.919566 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:14.919536    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": dial tcp 10.133.0.32:8000: connect: connection refused"
Sep 17 21:17:15.050698 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.050599    2582 generic.go:356] "Generic (PLEG): container finished" podID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerID="bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa" exitCode=0
Sep 17 21:17:15.050698 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.050651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerDied","Data":"bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa"}
Sep 17 21:17:15.167837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.167807    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:17:15.292255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292218    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292277    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2fpk\" (UniqueName: \"kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292401    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert\") pod \"c49969ad-c510-401b-ba11-29b2a26cfb4f\" (UID: \"c49969ad-c510-401b-ba11-29b2a26cfb4f\") "
Sep 17 21:17:15.292730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home" (OuterVolumeSpecName: "home") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.293087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292749    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.293087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.293087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:15.293087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.292863    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.294556 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.294529    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm" (OuterVolumeSpecName: "dshm") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.294679 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.294564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk" (OuterVolumeSpecName: "kube-api-access-p2fpk") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "kube-api-access-p2fpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:17:15.294679 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.294640    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:17:15.316114 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.316074    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c49969ad-c510-401b-ba11-29b2a26cfb4f" (UID: "c49969ad-c510-401b-ba11-29b2a26cfb4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.393806 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393763    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49969ad-c510-401b-ba11-29b2a26cfb4f-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.393806 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393793    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.393806 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393803    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fpk\" (UniqueName: \"kubernetes.io/projected/c49969ad-c510-401b-ba11-29b2a26cfb4f-kube-api-access-p2fpk\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.393806 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393811    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.394143 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393821    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.394143 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393829    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49969ad-c510-401b-ba11-29b2a26cfb4f-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.394143 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.393836    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49969ad-c510-401b-ba11-29b2a26cfb4f-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.437770 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.437744    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:17:15.594962 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.594939    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs\") pod \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") "
Sep 17 21:17:15.595065 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.595051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir\") pod \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") "
Sep 17 21:17:15.595115 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.595079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location\") pod \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") "
Sep 17 21:17:15.595154 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.595117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert\") pod \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\" (UID: \"a5c01db3-994d-43d3-b179-1bf4c3d2d404\") "
Sep 17 21:17:15.595443 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.595415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5c01db3-994d-43d3-b179-1bf4c3d2d404" (UID: "a5c01db3-994d-43d3-b179-1bf4c3d2d404"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.595505 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.595487    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5c01db3-994d-43d3-b179-1bf4c3d2d404" (UID: "a5c01db3-994d-43d3-b179-1bf4c3d2d404"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:15.597061 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.597040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5c01db3-994d-43d3-b179-1bf4c3d2d404" (UID: "a5c01db3-994d-43d3-b179-1bf4c3d2d404"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:17:15.615236 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.615204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5c01db3-994d-43d3-b179-1bf4c3d2d404" (UID: "a5c01db3-994d-43d3-b179-1bf4c3d2d404"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:15.696796 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.696758    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.696796 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.696797    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5c01db3-994d-43d3-b179-1bf4c3d2d404-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.696979 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.696813    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5c01db3-994d-43d3-b179-1bf4c3d2d404-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:15.696979 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:15.696828    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c01db3-994d-43d3-b179-1bf4c3d2d404-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:17:16.057686 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.057643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6" event={"ID":"c49969ad-c510-401b-ba11-29b2a26cfb4f","Type":"ContainerDied","Data":"672953be58df85ef6800d977254a1fa0e8a98209ef591f0143892841c960dc8d"}
Sep 17 21:17:16.057686 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.057684    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"
Sep 17 21:17:16.057911 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.057718    2582 scope.go:117] "RemoveContainer" containerID="bc7bfcc3066920b33eb326b8f84dc299569715f2b1c0194ad4cf66422b750daa"
Sep 17 21:17:16.059462 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.059431    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerID="8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d" exitCode=0
Sep 17 21:17:16.059574 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.059485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerDied","Data":"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"}
Sep 17 21:17:16.059574 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.059489    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"
Sep 17 21:17:16.059574 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.059514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865" event={"ID":"a5c01db3-994d-43d3-b179-1bf4c3d2d404","Type":"ContainerDied","Data":"afcc15ad7999e3393c4a127460039fbfbff31e3e9490cb5d1e596e1e29918e2d"}
Sep 17 21:17:16.067424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.067407    2582 scope.go:117] "RemoveContainer" containerID="201b1e900a4678a134d11691c0c44e93ff512df944c976a9b917aa5391b3b17b"
Sep 17 21:17:16.075179 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.075156    2582 scope.go:117] "RemoveContainer" containerID="8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"
Sep 17 21:17:16.078687 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.078666    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:17:16.082984 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.082965    2582 scope.go:117] "RemoveContainer" containerID="9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"
Sep 17 21:17:16.084440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.084419    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f478cf788-4rsc6"]
Sep 17 21:17:16.093309 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.093290    2582 scope.go:117] "RemoveContainer" containerID="8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"
Sep 17 21:17:16.093666 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:17:16.093642    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d\": container with ID starting with 8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d not found: ID does not exist" containerID="8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"
Sep 17 21:17:16.093730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.093667    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d"} err="failed to get container status \"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d\": rpc error: code = NotFound desc = could not find container \"8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d\": container with ID starting with 8e01944b57cc848468eab70178067861be2ca516197b6b8d53c2695da1dc6d9d not found: ID does not exist"
Sep 17 21:17:16.093730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.093684    2582 scope.go:117] "RemoveContainer" containerID="9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"
Sep 17 21:17:16.094006 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:17:16.093984    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df\": container with ID starting with 9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df not found: ID does not exist" containerID="9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"
Sep 17 21:17:16.094062 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.094016    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df"} err="failed to get container status \"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df\": rpc error: code = NotFound desc = could not find container \"9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df\": container with ID starting with 9a4f3c0553b42d3440dba215367237e588e67edbe5c988b5ef26cc91d33138df not found: ID does not exist"
Sep 17 21:17:16.094822 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.094804    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:17:16.098550 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:16.098526    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-68dbdb9fd6-dj865"]
Sep 17 21:17:17.584081 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:17.584050    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" path="/var/lib/kubelet/pods/a5c01db3-994d-43d3-b179-1bf4c3d2d404/volumes"
Sep 17 21:17:17.584575 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:17:17.584559    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" path="/var/lib/kubelet/pods/c49969ad-c510-401b-ba11-29b2a26cfb4f/volumes"
Sep 17 21:18:18.473052 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473016    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473339    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473350    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473371    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="storage-initializer"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473376    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="storage-initializer"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473387    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="main"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473392    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="main"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473400    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="storage-initializer"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473405    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="storage-initializer"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473460    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c01db3-994d-43d3-b179-1bf4c3d2d404" containerName="main"
Sep 17 21:18:18.473506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.473471    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49969ad-c510-401b-ba11-29b2a26cfb4f" containerName="main"
Sep 17 21:18:18.477816 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.477799    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.481035 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.481017    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 21:18:18.481602 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.481581    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-7lbzg\""
Sep 17 21:18:18.482848 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.482828    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:18:18.483931 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.483909    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 21:18:18.483931 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.483931    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 21:18:18.484154 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.484001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 21:18:18.512465 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.512432    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:18.519712 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519827 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519827 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519736    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519827 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519929 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519929 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.519929 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.520026 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.519930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm644\" (UniqueName: \"kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620734 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.620934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.620929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm644\" (UniqueName: \"kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.621385 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.621213    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.621385 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.621235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.621491 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.621468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.621532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.621484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.621728 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.621708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.623222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.623201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.623367 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.623353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.629037 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.629014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm644\" (UniqueName: \"kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644\") pod \"tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.763902 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.763875    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:18.768506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.768452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.784904 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.784876    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:18.787921 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.787897    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:18.822870 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.822835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.823011 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.822914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.823011 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.822933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.823011 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.822950    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924170 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924170 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924369 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924369 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924656 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.924770 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.924754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:18.926608 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:18.926589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:19.064028 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:19.063956    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:19.067631 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:18:19.067606    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691f2d5c_0ba3_44f3_8b0a_7b99a48e214f.slice/crio-ceb72921ba91cca232d1bb6a7aeb13e12cc0038f72dc02fe4a1136b5c42b51e5 WatchSource:0}: Error finding container ceb72921ba91cca232d1bb6a7aeb13e12cc0038f72dc02fe4a1136b5c42b51e5: Status 404 returned error can't find the container with id ceb72921ba91cca232d1bb6a7aeb13e12cc0038f72dc02fe4a1136b5c42b51e5
Sep 17 21:18:19.079928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:19.079910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:19.345402 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:19.345313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerStarted","Data":"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"}
Sep 17 21:18:19.345402 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:19.345371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerStarted","Data":"ceb72921ba91cca232d1bb6a7aeb13e12cc0038f72dc02fe4a1136b5c42b51e5"}
Sep 17 21:18:19.354233 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:19.354208    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:19.356783 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:18:19.356755    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898ad0e5_d6e9_4f08_9974_17fa60c41d95.slice/crio-ecc9744d7e3180811bf80135966799a40540bf83561bc1db75636c1b49ac842f WatchSource:0}: Error finding container ecc9744d7e3180811bf80135966799a40540bf83561bc1db75636c1b49ac842f: Status 404 returned error can't find the container with id ecc9744d7e3180811bf80135966799a40540bf83561bc1db75636c1b49ac842f
Sep 17 21:18:20.352989 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:20.352954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerStarted","Data":"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"}
Sep 17 21:18:20.353393 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:20.353010    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerStarted","Data":"ecc9744d7e3180811bf80135966799a40540bf83561bc1db75636c1b49ac842f"}
Sep 17 21:18:22.366280 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:22.366244    2582 generic.go:356] "Generic (PLEG): container finished" podID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerID="d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163" exitCode=0
Sep 17 21:18:22.366746 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:22.366312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerDied","Data":"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"}
Sep 17 21:18:22.368079 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:22.368053    2582 generic.go:356] "Generic (PLEG): container finished" podID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerID="d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7" exitCode=0
Sep 17 21:18:22.368186 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:22.368105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerDied","Data":"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"}
Sep 17 21:18:23.374549 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.374514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerStarted","Data":"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"}
Sep 17 21:18:23.375006 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.374868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:23.376552 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.376524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerStarted","Data":"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"}
Sep 17 21:18:23.394171 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.394126    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" podStartSLOduration=2.946570953 podStartE2EDuration="5.394112819s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.447541866s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:23.391880355 +0000 UTC m=+1753.971116205" watchObservedRunningTime="2026-09-17 21:18:23.394112819 +0000 UTC m=+1753.973348664"
Sep 17 21:18:23.410528 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.410484    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" podStartSLOduration=2.947265876 podStartE2EDuration="5.410469797s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.463203921s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:23.409569068 +0000 UTC m=+1753.988804920" watchObservedRunningTime="2026-09-17 21:18:23.410469797 +0000 UTC m=+1753.989705655"
Sep 17 21:18:23.788300 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:23.788267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:28.789000 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:28.788961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:28.805396 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:28.805361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:29.420123 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:29.420065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:54.391875 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:54.391790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:56.222337 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:56.222312    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm_898ad0e5-d6e9-4f08-9974-17fa60c41d95/main/0.log"
Sep 17 21:18:57.750116 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:57.750058    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:57.750603 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:57.750369    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="main" containerID="cri-o://9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d" gracePeriod=30
Sep 17 21:18:57.757011 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:57.756982    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:57.765563 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:57.765533    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:18:57.765563 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765563 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765563 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e" cmd=["/bin/sleep","15"]
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:57.765587    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" message="" err=<
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:57.765604    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="main" containerID="cri-o://3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"
Sep 17 21:18:57.765844 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:57.765645    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="main" containerID="cri-o://3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e" gracePeriod=60
Sep 17 21:18:58.119822 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.119795    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:58.177469 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177437    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177523    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177560    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177593    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177661    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177694    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm644\" (UniqueName: \"kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644\") pod \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\" (UID: \"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f\") "
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache" (OuterVolumeSpecName: "model-cache") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home" (OuterVolumeSpecName: "home") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.177851 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.177808    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:58.178193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.178016    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.178193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.178034    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.178193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.178048    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.178193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.178083    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.180256 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.180222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:58.180643 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.180607    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644" (OuterVolumeSpecName: "kube-api-access-tm644") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "kube-api-access-tm644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:58.181063 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.181029    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm" (OuterVolumeSpecName: "dshm") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.200764 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.200737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" (UID: "691f2d5c-0ba3-44f3-8b0a-7b99a48e214f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.278981 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.278949    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.279122 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.278987    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.279122 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.279002    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.279122 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.279013    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.279122 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.279028    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm644\" (UniqueName: \"kubernetes.io/projected/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f-kube-api-access-tm644\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.370175 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.370148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:58.379982 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.379959    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir\") pod \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") "
Sep 17 21:18:58.380141 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.380023    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location\") pod \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") "
Sep 17 21:18:58.380141 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.380067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert\") pod \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") "
Sep 17 21:18:58.380141 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.380087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs\") pod \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\" (UID: \"898ad0e5-d6e9-4f08-9974-17fa60c41d95\") "
Sep 17 21:18:58.380380 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.380359    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "898ad0e5-d6e9-4f08-9974-17fa60c41d95" (UID: "898ad0e5-d6e9-4f08-9974-17fa60c41d95"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:58.380453 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.380386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "898ad0e5-d6e9-4f08-9974-17fa60c41d95" (UID: "898ad0e5-d6e9-4f08-9974-17fa60c41d95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.382035 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.382015    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "898ad0e5-d6e9-4f08-9974-17fa60c41d95" (UID: "898ad0e5-d6e9-4f08-9974-17fa60c41d95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:58.401340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.401310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "898ad0e5-d6e9-4f08-9974-17fa60c41d95" (UID: "898ad0e5-d6e9-4f08-9974-17fa60c41d95"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:58.480950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.480922    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898ad0e5-d6e9-4f08-9974-17fa60c41d95-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.480950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.480947    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.480950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.480956    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.481188 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.480965    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898ad0e5-d6e9-4f08-9974-17fa60c41d95-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:18:58.553403 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.553366    2582 generic.go:356] "Generic (PLEG): container finished" podID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerID="9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d" exitCode=0
Sep 17 21:18:58.553569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.553419    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"
Sep 17 21:18:58.553569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.553458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerDied","Data":"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"}
Sep 17 21:18:58.553569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.553512    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm" event={"ID":"898ad0e5-d6e9-4f08-9974-17fa60c41d95","Type":"ContainerDied","Data":"ecc9744d7e3180811bf80135966799a40540bf83561bc1db75636c1b49ac842f"}
Sep 17 21:18:58.553569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.553534    2582 scope.go:117] "RemoveContainer" containerID="9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"
Sep 17 21:18:58.555351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.555314    2582 generic.go:356] "Generic (PLEG): container finished" podID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerID="3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e" exitCode=0
Sep 17 21:18:58.555480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.555371    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"
Sep 17 21:18:58.555480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.555382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerDied","Data":"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"}
Sep 17 21:18:58.555480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.555423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd" event={"ID":"691f2d5c-0ba3-44f3-8b0a-7b99a48e214f","Type":"ContainerDied","Data":"ceb72921ba91cca232d1bb6a7aeb13e12cc0038f72dc02fe4a1136b5c42b51e5"}
Sep 17 21:18:58.562461 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.562415    2582 scope.go:117] "RemoveContainer" containerID="d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"
Sep 17 21:18:58.569859 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.569842    2582 scope.go:117] "RemoveContainer" containerID="9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"
Sep 17 21:18:58.570139 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:58.570121    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d\": container with ID starting with 9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d not found: ID does not exist" containerID="9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"
Sep 17 21:18:58.570194 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.570148    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d"} err="failed to get container status \"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d\": rpc error: code = NotFound desc = could not find container \"9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d\": container with ID starting with 9e41dadf24895f170a5541b3160ee75bbe5747aa8604eb44fa899e5107fa655d not found: ID does not exist"
Sep 17 21:18:58.570194 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.570166    2582 scope.go:117] "RemoveContainer" containerID="d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"
Sep 17 21:18:58.570408 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:58.570391    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163\": container with ID starting with d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163 not found: ID does not exist" containerID="d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"
Sep 17 21:18:58.570466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.570417    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163"} err="failed to get container status \"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163\": rpc error: code = NotFound desc = could not find container \"d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163\": container with ID starting with d6f65291396f0b788ac41a2928d5487af096fc62647a0b211ebd0b0217223163 not found: ID does not exist"
Sep 17 21:18:58.570466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.570439    2582 scope.go:117] "RemoveContainer" containerID="3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"
Sep 17 21:18:58.578462 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.578445    2582 scope.go:117] "RemoveContainer" containerID="d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"
Sep 17 21:18:58.585456 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.585439    2582 scope.go:117] "RemoveContainer" containerID="3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"
Sep 17 21:18:58.585715 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:58.585699    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e\": container with ID starting with 3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e not found: ID does not exist" containerID="3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"
Sep 17 21:18:58.585763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.585720    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e"} err="failed to get container status \"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e\": rpc error: code = NotFound desc = could not find container \"3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e\": container with ID starting with 3ffc8ed07b10b6c9b39fbb12984df09de69da5b29ac647023acf227e44bee67e not found: ID does not exist"
Sep 17 21:18:58.585763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.585735    2582 scope.go:117] "RemoveContainer" containerID="d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"
Sep 17 21:18:58.585975 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:18:58.585959    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7\": container with ID starting with d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7 not found: ID does not exist" containerID="d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"
Sep 17 21:18:58.586030 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.585984    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7"} err="failed to get container status \"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7\": rpc error: code = NotFound desc = could not find container \"d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7\": container with ID starting with d61ee9ed59344086bc1fdfeddf5dd16bfb803c4f99f413724c22342ed9d775d7 not found: ID does not exist"
Sep 17 21:18:58.591561 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.591538    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:58.599282 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.599262    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7c86fb56dd-kd8dd"]
Sep 17 21:18:58.613164 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.613145    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:58.618149 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:58.618129    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-5b9b855b8b-kglwm"]
Sep 17 21:18:59.583678 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:59.583646    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" path="/var/lib/kubelet/pods/691f2d5c-0ba3-44f3-8b0a-7b99a48e214f/volumes"
Sep 17 21:18:59.584083 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:18:59.584070    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" path="/var/lib/kubelet/pods/898ad0e5-d6e9-4f08-9974-17fa60c41d95/volumes"
Sep 17 21:19:11.601840 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:11.601803    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:19:11.604698 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:11.604664    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:19:11.606695 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:11.606673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:19:11.609571 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:11.609557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:19:55.316957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.316923    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317290    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="storage-initializer"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317304    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="storage-initializer"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317315    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="main"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317323    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="main"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317334    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="storage-initializer"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317340    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="storage-initializer"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317347    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="main"
Sep 17 21:19:55.317399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317352    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="main"
Sep 17 21:19:55.317658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317418    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="691f2d5c-0ba3-44f3-8b0a-7b99a48e214f" containerName="main"
Sep 17 21:19:55.317658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.317430    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="898ad0e5-d6e9-4f08-9974-17fa60c41d95" containerName="main"
Sep 17 21:19:55.320450 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.320435    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.323509 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.323488    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8pwgf\""
Sep 17 21:19:55.327661 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.327641    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 21:19:55.327781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.327710    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 21:19:55.327934 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.327918    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 21:19:55.341613 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.341710 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.341710 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszcf\" (UniqueName: \"kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.341820 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.341820 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.341928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.341855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.343529 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.343510    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:19:55.442268 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442389 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442389 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszcf\" (UniqueName: \"kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442389 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442389 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442602 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442680 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442737 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.442778 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.442755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.444618 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.444602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.444885 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.444867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.451324 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.451296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszcf\" (UniqueName: \"kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.630392 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.630314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:19:55.917153 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:55.917129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:19:55.919231 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:19:55.919207    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e03750_161f_469b_a68f_c526e4764e48.slice/crio-91dbf1c009c966df14377b1d919bec185681ba7b64fd4dc55889ae17eb28fe73 WatchSource:0}: Error finding container 91dbf1c009c966df14377b1d919bec185681ba7b64fd4dc55889ae17eb28fe73: Status 404 returned error can't find the container with id 91dbf1c009c966df14377b1d919bec185681ba7b64fd4dc55889ae17eb28fe73
Sep 17 21:19:56.806161 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:56.806124    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" event={"ID":"91e03750-161f-469b-a68f-c526e4764e48","Type":"ContainerStarted","Data":"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"}
Sep 17 21:19:56.806561 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:56.806183    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" event={"ID":"91e03750-161f-469b-a68f-c526e4764e48","Type":"ContainerStarted","Data":"91dbf1c009c966df14377b1d919bec185681ba7b64fd4dc55889ae17eb28fe73"}
Sep 17 21:19:56.830701 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:56.830646    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" podStartSLOduration=1.830628998 podStartE2EDuration="1.830628998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:56.83028793 +0000 UTC m=+1847.409523785" watchObservedRunningTime="2026-09-17 21:19:56.830628998 +0000 UTC m=+1847.409864849"
Sep 17 21:19:57.631222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:19:57.631189    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:05.630861 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:05.630824    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:05.643227 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:05.643203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:05.855815 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:05.855778    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:56.615262 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:56.615228    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:20:56.623999 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:20:56.623959    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:20:56.623999 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:56.623999 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:56.623999 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa" cmd=["/bin/sleep","15"]
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:56.624024    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" message="" err=<
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:20:56.624041    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main" containerID="cri-o://a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"
Sep 17 21:20:56.624259 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:56.624141    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main" containerID="cri-o://a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa" gracePeriod=60
Sep 17 21:20:56.845697 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:56.845659    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Sep 17 21:20:56.974608 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:56.974583    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:57.070464 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dszcf\" (UniqueName: \"kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070515    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070624    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache\") pod \"91e03750-161f-469b-a68f-c526e4764e48\" (UID: \"91e03750-161f-469b-a68f-c526e4764e48\") "
Sep 17 21:20:57.070904 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home" (OuterVolumeSpecName: "home") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:57.070957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:57.070957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.070932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache" (OuterVolumeSpecName: "model-cache") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:57.072748 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.072717    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm" (OuterVolumeSpecName: "dshm") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:57.072862 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.072760    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:57.072862 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.072772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf" (OuterVolumeSpecName: "kube-api-access-dszcf") pod "91e03750-161f-469b-a68f-c526e4764e48" (UID: "91e03750-161f-469b-a68f-c526e4764e48"). InnerVolumeSpecName "kube-api-access-dszcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:57.080532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.080505    2582 generic.go:356] "Generic (PLEG): container finished" podID="91e03750-161f-469b-a68f-c526e4764e48" containerID="a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa" exitCode=0
Sep 17 21:20:57.080652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.080567    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"
Sep 17 21:20:57.080652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.080614    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" event={"ID":"91e03750-161f-469b-a68f-c526e4764e48","Type":"ContainerDied","Data":"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"}
Sep 17 21:20:57.080761 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.080673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9" event={"ID":"91e03750-161f-469b-a68f-c526e4764e48","Type":"ContainerDied","Data":"91dbf1c009c966df14377b1d919bec185681ba7b64fd4dc55889ae17eb28fe73"}
Sep 17 21:20:57.080761 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.080695    2582 scope.go:117] "RemoveContainer" containerID="a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"
Sep 17 21:20:57.090841 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.090825    2582 scope.go:117] "RemoveContainer" containerID="a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"
Sep 17 21:20:57.091136 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:20:57.091118    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa\": container with ID starting with a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa not found: ID does not exist" containerID="a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"
Sep 17 21:20:57.091194 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.091143    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa"} err="failed to get container status \"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa\": rpc error: code = NotFound desc = could not find container \"a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa\": container with ID starting with a5a4c8f96a3e4614e11d8996469bdf4a19469fb242796b1164b5248cb288f4fa not found: ID does not exist"
Sep 17 21:20:57.107065 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.107043    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:20:57.112868 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.112848    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-lc7h9"]
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171359    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171384    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171397    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e03750-161f-469b-a68f-c526e4764e48-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171405    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dszcf\" (UniqueName: \"kubernetes.io/projected/91e03750-161f-469b-a68f-c526e4764e48-kube-api-access-dszcf\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171414    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.171418 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.171421    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e03750-161f-469b-a68f-c526e4764e48-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:20:57.584657 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:20:57.584625    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e03750-161f-469b-a68f-c526e4764e48" path="/var/lib/kubelet/pods/91e03750-161f-469b-a68f-c526e4764e48/volumes"
Sep 17 21:21:21.321699 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.321666    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:21:21.322167 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.322058    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main"
Sep 17 21:21:21.322167 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.322072    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main"
Sep 17 21:21:21.322167 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.322149    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e03750-161f-469b-a68f-c526e4764e48" containerName="main"
Sep 17 21:21:21.326546 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.326525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.329390 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.329369    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 21:21:21.329508 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.329396    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 21:21:21.329508 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.329440    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 21:21:21.329844 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.329827    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 21:21:21.329904 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.329886    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-jqrgx\""
Sep 17 21:21:21.330069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.330056    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:21:21.344388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.344369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:21:21.477420 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcss\" (UniqueName: \"kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.477821 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.477629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.578943 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.578876    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcss\" (UniqueName: \"kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.578943 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.578906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.578943 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.578941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.578984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.579995 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.579971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.581392 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.581368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.581764 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.581745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.591010 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.590985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcss\" (UniqueName: \"kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.637149 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.637130    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:21:21.917481 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:21.917455    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:21:22.191974 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:22.191881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerStarted","Data":"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"}
Sep 17 21:21:22.191974 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:22.191946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerStarted","Data":"0655965a6fff0ec251a5c4b666df987655f2159e02d39b1199477af12305bbc3"}
Sep 17 21:21:24.203512 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:24.203481    2582 generic.go:356] "Generic (PLEG): container finished" podID="009eb3af-6d79-4979-9ecd-956235018836" containerID="87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58" exitCode=0
Sep 17 21:21:24.203846 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:24.203560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerDied","Data":"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"}
Sep 17 21:21:24.204901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:24.204887    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:21:53.367301 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:53.367214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerStarted","Data":"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"}
Sep 17 21:21:53.397881 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:53.397832    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podStartSLOduration=2.077694154 podStartE2EDuration="32.397816274s" totalImagesPullingTime="28.187493564s" totalInitContainerRuntime="2.132628556s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:53.395825677 +0000 UTC m=+1963.975061529" watchObservedRunningTime="2026-09-17 21:21:53.397816274 +0000 UTC m=+1963.977052147"
Sep 17 21:21:53.637961 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:21:53.637866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:22:01.637702 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:01.637655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:22:01.639063 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:01.639034    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:22:11.638029 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:11.637990    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:22:21.638575 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:21.638530    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:22:31.638395 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:31.638353    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:22:41.638197 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:41.638159    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:22:51.637931 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:22:51.637888    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:23:01.637963 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:01.637925    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:23:11.637867 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:11.637827    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:23:12.009368 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.009335    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:23:12.012958 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.012939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.016530 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.016498    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-wl8b6\""
Sep 17 21:23:12.016651 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.016511    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 21:23:12.016651 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.016576    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 21:23:12.016883 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.016862    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-6mjf7\""
Sep 17 21:23:12.016982 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.016888    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 21:23:12.022711 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.022668    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:23:12.082781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.082749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsgk\" (UniqueName: \"kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.082951 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.082801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.183845 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.183809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.184046 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.183905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsgk\" (UniqueName: \"kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.186597 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.186565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.192617 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.192589    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsgk\" (UniqueName: \"kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.324833 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.324751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.597304 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.597279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:23:12.600193 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:23:12.600161    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7c3b2e_d8c0_453c_8306_80483577ca82.slice/crio-472e91c92362b5b44ace47600cf8085062505f56cc0c19b1a670674e95b8d3c1 WatchSource:0}: Error finding container 472e91c92362b5b44ace47600cf8085062505f56cc0c19b1a670674e95b8d3c1: Status 404 returned error can't find the container with id 472e91c92362b5b44ace47600cf8085062505f56cc0c19b1a670674e95b8d3c1
Sep 17 21:23:12.745113 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.745040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" event={"ID":"8c7c3b2e-d8c0-453c-8306-80483577ca82","Type":"ContainerStarted","Data":"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"}
Sep 17 21:23:12.745562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.745140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" event={"ID":"8c7c3b2e-d8c0-453c-8306-80483577ca82","Type":"ContainerStarted","Data":"472e91c92362b5b44ace47600cf8085062505f56cc0c19b1a670674e95b8d3c1"}
Sep 17 21:23:12.745562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.745336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:12.764589 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:12.764540    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podStartSLOduration=1.7645270979999998 podStartE2EDuration="1.764527098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:23:12.76372874 +0000 UTC m=+2043.342964591" watchObservedRunningTime="2026-09-17 21:23:12.764527098 +0000 UTC m=+2043.343762947"
Sep 17 21:23:21.637900 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:21.637862    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:23:31.637818 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:31.637731    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 17 21:23:41.647581 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:41.647551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:23:41.655421 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:41.655391    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:23:43.752704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:43.752678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:23:45.237858 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:45.237822    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:23:48.012176 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:48.012141    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:23:53.752948 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:23:53.752904    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:00.262974 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:00.262918    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main" containerID="cri-o://adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0" gracePeriod=45
Sep 17 21:24:03.039826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.039791    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" containerID="cri-o://46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426" gracePeriod=45
Sep 17 21:24:03.708279 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.708255    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:24:03.752693 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.752654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:03.839413 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839335    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839413 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839378    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839631 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839435    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839631 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839457    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839631 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839496    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbcss\" (UniqueName: \"kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839631 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839568    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.839837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839741    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache" (OuterVolumeSpecName: "model-cache") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:03.839950 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.839833    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:03.840244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.840218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home" (OuterVolumeSpecName: "home") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:03.840358 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.840282    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location\") pod \"009eb3af-6d79-4979-9ecd-956235018836\" (UID: \"009eb3af-6d79-4979-9ecd-956235018836\") "
Sep 17 21:24:03.840565 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.840547    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.840634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.840570    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/009eb3af-6d79-4979-9ecd-956235018836-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.840634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.840586    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.841859 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.841814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm" (OuterVolumeSpecName: "dshm") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:03.841959 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.841918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:03.841959 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.841944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss" (OuterVolumeSpecName: "kube-api-access-bbcss") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "kube-api-access-bbcss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:03.851400 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.851372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:03.863558 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.863534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "009eb3af-6d79-4979-9ecd-956235018836" (UID: "009eb3af-6d79-4979-9ecd-956235018836"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:03.941843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.941816    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.941843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.941838    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbcss\" (UniqueName: \"kubernetes.io/projected/009eb3af-6d79-4979-9ecd-956235018836-kube-api-access-bbcss\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.941993 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.941848    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/009eb3af-6d79-4979-9ecd-956235018836-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.941993 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.941858    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.941993 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.941866    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/009eb3af-6d79-4979-9ecd-956235018836-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:03.983793 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.983761    2582 generic.go:356] "Generic (PLEG): container finished" podID="009eb3af-6d79-4979-9ecd-956235018836" containerID="46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426" exitCode=0
Sep 17 21:24:03.983924 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.983808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerDied","Data":"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"}
Sep 17 21:24:03.983924 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.983863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" event={"ID":"009eb3af-6d79-4979-9ecd-956235018836","Type":"ContainerDied","Data":"0655965a6fff0ec251a5c4b666df987655f2159e02d39b1199477af12305bbc3"}
Sep 17 21:24:03.983924 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.983812    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"
Sep 17 21:24:03.983924 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.983892    2582 scope.go:117] "RemoveContainer" containerID="46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"
Sep 17 21:24:03.993025 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:03.993007    2582 scope.go:117] "RemoveContainer" containerID="87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"
Sep 17 21:24:04.000466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.000450    2582 scope.go:117] "RemoveContainer" containerID="46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"
Sep 17 21:24:04.000728 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:24:04.000709    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426\": container with ID starting with 46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426 not found: ID does not exist" containerID="46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"
Sep 17 21:24:04.000777 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.000735    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426"} err="failed to get container status \"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426\": rpc error: code = NotFound desc = could not find container \"46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426\": container with ID starting with 46984d282182196741254fbf9933cc38588c02b7868d80dcc85a6d8a33ac0426 not found: ID does not exist"
Sep 17 21:24:04.000777 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.000752    2582 scope.go:117] "RemoveContainer" containerID="87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"
Sep 17 21:24:04.000963 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:24:04.000948    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58\": container with ID starting with 87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58 not found: ID does not exist" containerID="87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"
Sep 17 21:24:04.001007 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.000969    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58"} err="failed to get container status \"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58\": rpc error: code = NotFound desc = could not find container \"87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58\": container with ID starting with 87f9d4d6a6b9482ef0967575fbf0b236d182bfe7188075c0884780d057ceea58 not found: ID does not exist"
Sep 17 21:24:04.004998 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.004979    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:24:04.016733 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.016707    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2"]
Sep 17 21:24:04.648520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:04.648483    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b76c7f7f-d5sr2" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 21:24:05.583734 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:05.583691    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009eb3af-6d79-4979-9ecd-956235018836" path="/var/lib/kubelet/pods/009eb3af-6d79-4979-9ecd-956235018836/volumes"
Sep 17 21:24:11.630504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:11.630476    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:24:11.634545 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:11.634522    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:24:11.635228 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:11.635211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:24:11.639302 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:11.639287    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:24:13.752677 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:13.752633    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:13.753158 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:13.752730    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:24:23.752776 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:23.752734    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:24:30.508597 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.508571    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:24:30.565724 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.565651    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxsgk\" (UniqueName: \"kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk\") pod \"8c7c3b2e-d8c0-453c-8306-80483577ca82\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") "
Sep 17 21:24:30.565724 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.565694    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs\") pod \"8c7c3b2e-d8c0-453c-8306-80483577ca82\" (UID: \"8c7c3b2e-d8c0-453c-8306-80483577ca82\") "
Sep 17 21:24:30.567955 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.567933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c7c3b2e-d8c0-453c-8306-80483577ca82" (UID: "8c7c3b2e-d8c0-453c-8306-80483577ca82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:30.568045 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.567972    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk" (OuterVolumeSpecName: "kube-api-access-hxsgk") pod "8c7c3b2e-d8c0-453c-8306-80483577ca82" (UID: "8c7c3b2e-d8c0-453c-8306-80483577ca82"). InnerVolumeSpecName "kube-api-access-hxsgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:30.666424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.666390    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsgk\" (UniqueName: \"kubernetes.io/projected/8c7c3b2e-d8c0-453c-8306-80483577ca82-kube-api-access-hxsgk\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:30.666424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:30.666424    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7c3b2e-d8c0-453c-8306-80483577ca82-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:24:31.100561 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.100523    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerID="adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0" exitCode=0
Sep 17 21:24:31.100755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.100625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" event={"ID":"8c7c3b2e-d8c0-453c-8306-80483577ca82","Type":"ContainerDied","Data":"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"}
Sep 17 21:24:31.100755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.100642    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"
Sep 17 21:24:31.100755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.100703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq" event={"ID":"8c7c3b2e-d8c0-453c-8306-80483577ca82","Type":"ContainerDied","Data":"472e91c92362b5b44ace47600cf8085062505f56cc0c19b1a670674e95b8d3c1"}
Sep 17 21:24:31.100755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.100731    2582 scope.go:117] "RemoveContainer" containerID="adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"
Sep 17 21:24:31.109537 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.109520    2582 scope.go:117] "RemoveContainer" containerID="adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"
Sep 17 21:24:31.109930 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:24:31.109903    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0\": container with ID starting with adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0 not found: ID does not exist" containerID="adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"
Sep 17 21:24:31.110031 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.109934    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0"} err="failed to get container status \"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0\": rpc error: code = NotFound desc = could not find container \"adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0\": container with ID starting with adfe844009d8270f232b90c88f51c560483092b3dada925d324029052ecef6d0 not found: ID does not exist"
Sep 17 21:24:31.123730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.123705    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:24:31.127935 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.127911    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schep28zq"]
Sep 17 21:24:31.585443 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:31.585410    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" path="/var/lib/kubelet/pods/8c7c3b2e-d8c0-453c-8306-80483577ca82/volumes"
Sep 17 21:24:48.495703 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.495661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496011    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="storage-initializer"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496023    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="storage-initializer"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496032    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496037    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496053    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496061    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496147    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="009eb3af-6d79-4979-9ecd-956235018836" containerName="main"
Sep 17 21:24:48.496320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.496155    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7c3b2e-d8c0-453c-8306-80483577ca82" containerName="main"
Sep 17 21:24:48.499317 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.499300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.501653 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.501628    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-6tf4r\""
Sep 17 21:24:48.501765 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.501630    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 21:24:48.501960 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.501947    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 21:24:48.502024 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.502010    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 21:24:48.515953 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.515932    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:24:48.611409 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611409 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611551    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611612 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.611779 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.611616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqc5\" (UniqueName: \"kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712218 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqc5\" (UniqueName: \"kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712581 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712581 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712687 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712777 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.712947 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.713057 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.712986    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.715011 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.714985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.715193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.715174    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.725281 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.725259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqc5\" (UniqueName: \"kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:48.809601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:48.809571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:24:49.088316 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:49.088211    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:24:49.181958 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:49.181931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerStarted","Data":"ff9866dbb750b85d6601c5bcf5d2b013a421b8c2e9d53f618796009b05050adc"}
Sep 17 21:24:57.224772 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:57.224735    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerStarted","Data":"72cfad56225e208473c0a886a2d0d46859b18c90ca1080f28b950b50a5f42eb9"}
Sep 17 21:24:58.230756 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:58.230718    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerID="72cfad56225e208473c0a886a2d0d46859b18c90ca1080f28b950b50a5f42eb9" exitCode=0
Sep 17 21:24:58.231142 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:58.230802    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerDied","Data":"72cfad56225e208473c0a886a2d0d46859b18c90ca1080f28b950b50a5f42eb9"}
Sep 17 21:24:59.236858 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:59.236768    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerStarted","Data":"45bdd33dbb0cb8e271b827d0d2f9871a1f7738ad18f4466d3080175de38e8a9f"}
Sep 17 21:24:59.236858 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:59.236836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerStarted","Data":"b87ee53c4fb510f1233042e2dcdfc98fa67571f3ec4a6f759a6bea9089d36e38"}
Sep 17 21:24:59.261687 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:59.261642    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podStartSLOduration=2.964263576 podStartE2EDuration="11.261625635s" totalImagesPullingTime="7.211634762s" totalInitContainerRuntime="1.085727297s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:24:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:24:59.260555981 +0000 UTC m=+2149.839791831" watchObservedRunningTime="2026-09-17 21:24:59.261625635 +0000 UTC m=+2149.840861532"
Sep 17 21:24:59.810555 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:24:59.810514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:25:08.810172 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:08.810139    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:25:08.811810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:08.811781    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:25:18.810882 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:18.810840    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:25:28.810787 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:28.810750    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:25:38.810532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:38.810492    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:25:48.809955 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:48.809913    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:25:58.810755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:25:58.810710    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:26:08.810504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:08.810466    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:26:18.810705 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:18.810665    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:26:20.051759 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.051726    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:26:20.054271 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.054253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.057266 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.057237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 21:26:20.058139 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.058114    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 21:26:20.058270 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.058123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-wv2r8\""
Sep 17 21:26:20.058270 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.058176    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 21:26:20.058270 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.058237    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-5pmv8\""
Sep 17 21:26:20.067143 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.067119    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:26:20.173056 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.173018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.173056 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.173062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh9qt\" (UniqueName: \"kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.273464 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.273428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.273646 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.273472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh9qt\" (UniqueName: \"kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.275975 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.275952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.282798 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.282772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh9qt\" (UniqueName: \"kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.365310 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.365217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:20.640430 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:20.640403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:26:20.643123 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:26:20.643084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037c2b91_e7dd_457e_a729_fe917cbd6932.slice/crio-e67262a0d196215f8360ca550fbb8f4316dd13a67bc0b87b3907c7bc6d1d8d6d WatchSource:0}: Error finding container e67262a0d196215f8360ca550fbb8f4316dd13a67bc0b87b3907c7bc6d1d8d6d: Status 404 returned error can't find the container with id e67262a0d196215f8360ca550fbb8f4316dd13a67bc0b87b3907c7bc6d1d8d6d
Sep 17 21:26:21.635461 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:21.635421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" event={"ID":"037c2b91-e7dd-457e-a729-fe917cbd6932","Type":"ContainerStarted","Data":"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"}
Sep 17 21:26:21.635841 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:21.635482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" event={"ID":"037c2b91-e7dd-457e-a729-fe917cbd6932","Type":"ContainerStarted","Data":"e67262a0d196215f8360ca550fbb8f4316dd13a67bc0b87b3907c7bc6d1d8d6d"}
Sep 17 21:26:21.635841 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:21.635530    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:21.661027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:21.660975    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podStartSLOduration=1.660958816 podStartE2EDuration="1.660958816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:26:21.658876566 +0000 UTC m=+2232.238112418" watchObservedRunningTime="2026-09-17 21:26:21.660958816 +0000 UTC m=+2232.240194666"
Sep 17 21:26:28.810114 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:28.810009    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:26:38.811088 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:38.811045    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:26:48.820634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:48.820604    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:26:48.829236 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:48.829212    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:26:52.641957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:52.641921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:26:54.268324 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.268290    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:26:54.268749 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.268557    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar" containerID="cri-o://45bdd33dbb0cb8e271b827d0d2f9871a1f7738ad18f4466d3080175de38e8a9f" gracePeriod=60
Sep 17 21:26:54.630273 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.630179    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:26:54.805906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.805878    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq_fc701ae8-fe39-4529-81e4-e27d0a8f88e3/modelcar/0.log"
Sep 17 21:26:54.806535 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.806506    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerID="45bdd33dbb0cb8e271b827d0d2f9871a1f7738ad18f4466d3080175de38e8a9f" exitCode=143
Sep 17 21:26:54.806683 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:26:54.806579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerDied","Data":"45bdd33dbb0cb8e271b827d0d2f9871a1f7738ad18f4466d3080175de38e8a9f"}
Sep 17 21:27:02.641486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:02.641448    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:09.290958 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.290890    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" containerID="cri-o://b87ee53c4fb510f1233042e2dcdfc98fa67571f3ec4a6f759a6bea9089d36e38" gracePeriod=45
Sep 17 21:27:09.649747 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.649656    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main" containerID="cri-o://7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3" gracePeriod=45
Sep 17 21:27:09.821911 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.821871    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
Sep 17 21:27:09.877329 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.877304    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq_fc701ae8-fe39-4529-81e4-e27d0a8f88e3/modelcar/0.log"
Sep 17 21:27:09.877986 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.877966    2582 generic.go:356] "Generic (PLEG): container finished" podID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerID="b87ee53c4fb510f1233042e2dcdfc98fa67571f3ec4a6f759a6bea9089d36e38" exitCode=0
Sep 17 21:27:09.878074 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:09.878039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerDied","Data":"b87ee53c4fb510f1233042e2dcdfc98fa67571f3ec4a6f759a6bea9089d36e38"}
Sep 17 21:27:10.542014 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.541994    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq_fc701ae8-fe39-4529-81e4-e27d0a8f88e3/modelcar/0.log"
Sep 17 21:27:10.543971 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.543954    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:27:10.580989 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.580932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.580989 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.580983    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581036    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjqc5\" (UniqueName: \"kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581103    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs\") pod \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\" (UID: \"fc701ae8-fe39-4529-81e4-e27d0a8f88e3\") "
Sep 17 21:27:10.581454 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581354    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:10.581454 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581416    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.581772 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581603    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:10.581772 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.581755    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home" (OuterVolumeSpecName: "home") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:10.583758 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.583725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm" (OuterVolumeSpecName: "dshm") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:10.583848 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.583766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:10.583848 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.583808    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5" (OuterVolumeSpecName: "kube-api-access-gjqc5") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "kube-api-access-gjqc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:10.592840 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.592804    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc701ae8-fe39-4529-81e4-e27d0a8f88e3" (UID: "fc701ae8-fe39-4529-81e4-e27d0a8f88e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:10.682345 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682318    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqc5\" (UniqueName: \"kubernetes.io/projected/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kube-api-access-gjqc5\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.682345 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682341    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.682492 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682352    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.682492 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682360    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.682492 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682369    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.682492 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.682377    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc701ae8-fe39-4529-81e4-e27d0a8f88e3-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:10.884711 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.884632    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq_fc701ae8-fe39-4529-81e4-e27d0a8f88e3/modelcar/0.log"
Sep 17 21:27:10.885404 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.885375    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq" event={"ID":"fc701ae8-fe39-4529-81e4-e27d0a8f88e3","Type":"ContainerDied","Data":"ff9866dbb750b85d6601c5bcf5d2b013a421b8c2e9d53f618796009b05050adc"}
Sep 17 21:27:10.885546 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.885410    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"
Sep 17 21:27:10.885546 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.885479    2582 scope.go:117] "RemoveContainer" containerID="45bdd33dbb0cb8e271b827d0d2f9871a1f7738ad18f4466d3080175de38e8a9f"
Sep 17 21:27:10.894569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.894553    2582 scope.go:117] "RemoveContainer" containerID="b87ee53c4fb510f1233042e2dcdfc98fa67571f3ec4a6f759a6bea9089d36e38"
Sep 17 21:27:10.901865 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.901847    2582 scope.go:117] "RemoveContainer" containerID="72cfad56225e208473c0a886a2d0d46859b18c90ca1080f28b950b50a5f42eb9"
Sep 17 21:27:10.907514 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.907492    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:27:10.911266 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:10.911245    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d82jbq"]
Sep 17 21:27:11.585990 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:11.585955    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" path="/var/lib/kubelet/pods/fc701ae8-fe39-4529-81e4-e27d0a8f88e3/volumes"
Sep 17 21:27:12.641546 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:12.641513    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:22.641179 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:22.641145    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:22.641622 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:22.641246    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:27:32.641946 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:32.641911    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:27:39.893133 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:39.893109    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:27:40.015441 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.015405    2582 generic.go:356] "Generic (PLEG): container finished" podID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerID="7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3" exitCode=0
Sep 17 21:27:40.015441 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.015443    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"
Sep 17 21:27:40.015700 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.015450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" event={"ID":"037c2b91-e7dd-457e-a729-fe917cbd6932","Type":"ContainerDied","Data":"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"}
Sep 17 21:27:40.015700 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.015537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c" event={"ID":"037c2b91-e7dd-457e-a729-fe917cbd6932","Type":"ContainerDied","Data":"e67262a0d196215f8360ca550fbb8f4316dd13a67bc0b87b3907c7bc6d1d8d6d"}
Sep 17 21:27:40.015700 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.015568    2582 scope.go:117] "RemoveContainer" containerID="7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"
Sep 17 21:27:40.023327 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.023310    2582 scope.go:117] "RemoveContainer" containerID="7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"
Sep 17 21:27:40.023602 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:27:40.023583    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3\": container with ID starting with 7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3 not found: ID does not exist" containerID="7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"
Sep 17 21:27:40.023667 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.023611    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3"} err="failed to get container status \"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3\": rpc error: code = NotFound desc = could not find container \"7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3\": container with ID starting with 7a4b3d2c50266896a62f7ba4c3e86e22b7e878a604794745705b2a6d255c7de3 not found: ID does not exist"
Sep 17 21:27:40.031649 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.031628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs\") pod \"037c2b91-e7dd-457e-a729-fe917cbd6932\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") "
Sep 17 21:27:40.031754 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.031673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh9qt\" (UniqueName: \"kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt\") pod \"037c2b91-e7dd-457e-a729-fe917cbd6932\" (UID: \"037c2b91-e7dd-457e-a729-fe917cbd6932\") "
Sep 17 21:27:40.033594 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.033573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "037c2b91-e7dd-457e-a729-fe917cbd6932" (UID: "037c2b91-e7dd-457e-a729-fe917cbd6932"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:40.033727 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.033677    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt" (OuterVolumeSpecName: "kube-api-access-bh9qt") pod "037c2b91-e7dd-457e-a729-fe917cbd6932" (UID: "037c2b91-e7dd-457e-a729-fe917cbd6932"). InnerVolumeSpecName "kube-api-access-bh9qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:40.133132 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.133086    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037c2b91-e7dd-457e-a729-fe917cbd6932-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:40.133132 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.133128    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh9qt\" (UniqueName: \"kubernetes.io/projected/037c2b91-e7dd-457e-a729-fe917cbd6932-kube-api-access-bh9qt\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:27:40.336004 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.335972    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:27:40.344060 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:40.344034    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheftq5c"]
Sep 17 21:27:41.584882 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:41.584851    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" path="/var/lib/kubelet/pods/037c2b91-e7dd-457e-a729-fe917cbd6932/volumes"
Sep 17 21:27:52.251596 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251561    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251875    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251885    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251900    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar-init"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251906    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar-init"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251930    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251936    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251949    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main"
Sep 17 21:27:52.251976 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.251954    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main"
Sep 17 21:27:52.252248 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.252001    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="main"
Sep 17 21:27:52.252248 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.252011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="037c2b91-e7dd-457e-a729-fe917cbd6932" containerName="main"
Sep 17 21:27:52.252248 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.252017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc701ae8-fe39-4529-81e4-e27d0a8f88e3" containerName="modelcar"
Sep 17 21:27:52.256376 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.256359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.260054 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.260035    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 21:27:52.260967 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.260946    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 21:27:52.261164 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.261129    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 21:27:52.261240 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.261156    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-95c77\""
Sep 17 21:27:52.261464 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.261444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 21:27:52.261548 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.261520    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:27:52.275814 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.275792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:27:52.333142 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333291 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333291 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333373 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333373 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.333440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.333429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8jf\" (UniqueName: \"kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434004 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.433976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434183 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434183 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434183 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8jf\" (UniqueName: \"kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434681 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434754 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.434914 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.434896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.436651 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.436631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.436901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.436886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.442929 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.442906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8jf\" (UniqueName: \"kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf\") pod \"custom-route-timeout-test-kserve-7c6655d786-4pm94\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.566718 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.566633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:27:52.868993 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:52.868964    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:27:52.871699 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:27:52.871670    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7edbc76_c10f_46d3_bc6d_812d078082e8.slice/crio-1674211e116754e7ea161c896c4c58507452dc42b44813245f75a264b86820fb WatchSource:0}: Error finding container 1674211e116754e7ea161c896c4c58507452dc42b44813245f75a264b86820fb: Status 404 returned error can't find the container with id 1674211e116754e7ea161c896c4c58507452dc42b44813245f75a264b86820fb
Sep 17 21:27:53.074494 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:53.074459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerStarted","Data":"b74839c27b74cca5dbd3ce16e504cd94c1e2deccbd397d7e9633e4b720acd349"}
Sep 17 21:27:53.074713 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:53.074510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerStarted","Data":"1674211e116754e7ea161c896c4c58507452dc42b44813245f75a264b86820fb"}
Sep 17 21:27:55.085197 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:55.085168    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerID="b74839c27b74cca5dbd3ce16e504cd94c1e2deccbd397d7e9633e4b720acd349" exitCode=0
Sep 17 21:27:55.085602 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:55.085245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerDied","Data":"b74839c27b74cca5dbd3ce16e504cd94c1e2deccbd397d7e9633e4b720acd349"}
Sep 17 21:27:56.098936 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:56.098878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerStarted","Data":"7fbb3971f491c122c8a53d76701af4bd8ebe89ea040c44b63805051d3347de2c"}
Sep 17 21:27:56.129560 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:56.129496    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podStartSLOduration=2.046202451 podStartE2EDuration="4.129477763s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.083275312s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:27:56.127198443 +0000 UTC m=+2326.706434294" watchObservedRunningTime="2026-09-17 21:27:56.129477763 +0000 UTC m=+2326.708713614"
Sep 17 21:27:56.567533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:27:56.567501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:28:02.567620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:02.567581    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:28:02.569040 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:02.569009    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:28:12.567713 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:12.567672    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:28:22.568014 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:22.567970    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:28:32.567884 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:32.567836    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:28:42.567360 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:42.567301    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:28:52.567185 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:28:52.567144    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:02.568027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:02.567986    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:11.665085 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:11.665051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:29:11.670225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:11.670203    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:29:11.671504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:11.671483    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:29:11.676355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:11.676336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:29:12.567914 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:12.567862    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:22.567620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:22.567569    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:32.567258 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:32.567161    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:42.567747 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:42.567699    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:29:52.577082 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:52.577048    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:29:52.584772 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:52.584719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:29:56.975082 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:29:56.975046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:30:11.997906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:11.997857    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" containerID="cri-o://7fbb3971f491c122c8a53d76701af4bd8ebe89ea040c44b63805051d3347de2c" gracePeriod=45
Sep 17 21:30:12.578623 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.578581    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 21:30:12.730403 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.730367    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerID="7fbb3971f491c122c8a53d76701af4bd8ebe89ea040c44b63805051d3347de2c" exitCode=0
Sep 17 21:30:12.730572 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.730447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerDied","Data":"7fbb3971f491c122c8a53d76701af4bd8ebe89ea040c44b63805051d3347de2c"}
Sep 17 21:30:12.758161 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.758046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:30:12.762871 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.762851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.762883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.762909    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.762934    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng8jf\" (UniqueName: \"kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763027 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763024    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763281 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763055    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763281 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763090    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763281 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir\") pod \"a7edbc76-c10f-46d3-bc6d-812d078082e8\" (UID: \"a7edbc76-c10f-46d3-bc6d-812d078082e8\") "
Sep 17 21:30:12.763644 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763485    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:12.763644 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763575    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:30:12.763798 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.763706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home" (OuterVolumeSpecName: "home") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:12.765447 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.765407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf" (OuterVolumeSpecName: "kube-api-access-ng8jf") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "kube-api-access-ng8jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:30:12.765569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.765450    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm" (OuterVolumeSpecName: "dshm") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:12.765569 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.765486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:30:12.775209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.775181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:12.785327 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.785300    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7edbc76-c10f-46d3-bc6d-812d078082e8" (UID: "a7edbc76-c10f-46d3-bc6d-812d078082e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:30:12.864069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864033    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7edbc76-c10f-46d3-bc6d-812d078082e8-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864062    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864071    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864080    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8jf\" (UniqueName: \"kubernetes.io/projected/a7edbc76-c10f-46d3-bc6d-812d078082e8-kube-api-access-ng8jf\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864089    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864110    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7edbc76-c10f-46d3-bc6d-812d078082e8-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864117    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:12.864361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:12.864125    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7edbc76-c10f-46d3-bc6d-812d078082e8-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:30:13.737579 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.737540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"
Sep 17 21:30:13.737579 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.737559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94" event={"ID":"a7edbc76-c10f-46d3-bc6d-812d078082e8","Type":"ContainerDied","Data":"1674211e116754e7ea161c896c4c58507452dc42b44813245f75a264b86820fb"}
Sep 17 21:30:13.738044 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.737645    2582 scope.go:117] "RemoveContainer" containerID="7fbb3971f491c122c8a53d76701af4bd8ebe89ea040c44b63805051d3347de2c"
Sep 17 21:30:13.746241 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.746222    2582 scope.go:117] "RemoveContainer" containerID="b74839c27b74cca5dbd3ce16e504cd94c1e2deccbd397d7e9633e4b720acd349"
Sep 17 21:30:13.759192 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.759169    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:30:13.764712 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:13.764690    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7c6655d786-4pm94"]
Sep 17 21:30:15.586746 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:30:15.586633    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" path="/var/lib/kubelet/pods/a7edbc76-c10f-46d3-bc6d-812d078082e8/volumes"
Sep 17 21:31:09.705370 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705340    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:31:09.705810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705673    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main"
Sep 17 21:31:09.705810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705684    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main"
Sep 17 21:31:09.705810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="storage-initializer"
Sep 17 21:31:09.705810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705697    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="storage-initializer"
Sep 17 21:31:09.705810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.705744    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7edbc76-c10f-46d3-bc6d-812d078082e8" containerName="main"
Sep 17 21:31:09.708619 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.708601    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.720538 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.720518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 21:31:09.720660 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.720629    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-tq67m\""
Sep 17 21:31:09.720709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.720667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 21:31:09.720709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.720687    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 21:31:09.720823 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.720667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-bcz9h\""
Sep 17 21:31:09.730150 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.730124    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:31:09.742014 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.741981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.742163 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.742039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vnn\" (UniqueName: \"kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.842825 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.842788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.842978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.842861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vnn\" (UniqueName: \"kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.845427 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.845406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:09.855128 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:09.855077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vnn\" (UniqueName: \"kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:10.018230 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:10.018202    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:10.295555 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:10.295478    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:31:10.298867 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:31:10.298831    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fb33d3_d6c4_43f5_bc24_39eaa170b8c7.slice/crio-4f98ffb6b81063a10c5d426adc0c3f0a0b328649c1f27ed6f7512255bc253bfa WatchSource:0}: Error finding container 4f98ffb6b81063a10c5d426adc0c3f0a0b328649c1f27ed6f7512255bc253bfa: Status 404 returned error can't find the container with id 4f98ffb6b81063a10c5d426adc0c3f0a0b328649c1f27ed6f7512255bc253bfa
Sep 17 21:31:10.991320 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:10.991283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" event={"ID":"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7","Type":"ContainerStarted","Data":"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"}
Sep 17 21:31:10.991710 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:10.991337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" event={"ID":"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7","Type":"ContainerStarted","Data":"4f98ffb6b81063a10c5d426adc0c3f0a0b328649c1f27ed6f7512255bc253bfa"}
Sep 17 21:31:10.991710 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:10.991400    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:31:11.028269 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:11.028221    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podStartSLOduration=2.028205573 podStartE2EDuration="2.028205573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:31:11.023829021 +0000 UTC m=+2521.603064872" watchObservedRunningTime="2026-09-17 21:31:11.028205573 +0000 UTC m=+2521.607441424"
Sep 17 21:31:41.997913 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:31:41.997878    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:33:15.265178 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:15.265138    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:33:21.997763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:21.997720    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:33:30.292703 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:30.292653    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main" containerID="cri-o://dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b" gracePeriod=45
Sep 17 21:33:31.997621 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:31.997584    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:33:41.997324 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:41.997287    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:33:41.997697 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:41.997385    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:33:51.997590 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:33:51.997552    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:34:00.538939 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.538911    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:34:00.648087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.647977    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs\") pod \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") "
Sep 17 21:34:00.648087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.648066    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2vnn\" (UniqueName: \"kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn\") pod \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\" (UID: \"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7\") "
Sep 17 21:34:00.650421 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.650385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" (UID: "45fb33d3-d6c4-43f5-bc24-39eaa170b8c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:34:00.650550 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.650445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn" (OuterVolumeSpecName: "kube-api-access-h2vnn") pod "45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" (UID: "45fb33d3-d6c4-43f5-bc24-39eaa170b8c7"). InnerVolumeSpecName "kube-api-access-h2vnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:34:00.749055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.749023    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vnn\" (UniqueName: \"kubernetes.io/projected/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-kube-api-access-h2vnn\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:34:00.749055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.749048    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:34:00.752901 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.752875    2582 generic.go:356] "Generic (PLEG): container finished" podID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerID="dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b" exitCode=0
Sep 17 21:34:00.753052 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.752936    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"
Sep 17 21:34:00.753052 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.752942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" event={"ID":"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7","Type":"ContainerDied","Data":"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"}
Sep 17 21:34:00.753164 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.753052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4" event={"ID":"45fb33d3-d6c4-43f5-bc24-39eaa170b8c7","Type":"ContainerDied","Data":"4f98ffb6b81063a10c5d426adc0c3f0a0b328649c1f27ed6f7512255bc253bfa"}
Sep 17 21:34:00.753164 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.753074    2582 scope.go:117] "RemoveContainer" containerID="dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"
Sep 17 21:34:00.762222 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.762197    2582 scope.go:117] "RemoveContainer" containerID="dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"
Sep 17 21:34:00.762558 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:34:00.762538    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b\": container with ID starting with dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b not found: ID does not exist" containerID="dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"
Sep 17 21:34:00.762643 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.762567    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b"} err="failed to get container status \"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b\": rpc error: code = NotFound desc = could not find container \"dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b\": container with ID starting with dcf8aadea787eaf3ba1373e70d4897000be39ad4e19dfd5e9cff09f908f95b9b not found: ID does not exist"
Sep 17 21:34:00.774180 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.774144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:34:00.780339 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:00.780308    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-g55s4"]
Sep 17 21:34:01.583716 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:01.583677    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" path="/var/lib/kubelet/pods/45fb33d3-d6c4-43f5-bc24-39eaa170b8c7/volumes"
Sep 17 21:34:11.693383 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:11.693353    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:34:11.698008 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:11.697984    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:34:11.700119 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:11.700084    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:34:11.704440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:11.704421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:34:13.864902 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.864867    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:34:13.865293 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.865238    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main"
Sep 17 21:34:13.865293 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.865251    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main"
Sep 17 21:34:13.865372 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.865322    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fb33d3-d6c4-43f5-bc24-39eaa170b8c7" containerName="main"
Sep 17 21:34:13.869508 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.869483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.872022 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.871995    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 21:34:13.872022 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.872013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 21:34:13.872244 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.871995    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:34:13.872301 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.872287    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 21:34:13.872728 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.872709    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-wmnwr\""
Sep 17 21:34:13.872863 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.872716    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-kgfc2\""
Sep 17 21:34:13.872863 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.872784    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 21:34:13.886873 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.886848    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:34:13.949917 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.949882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.949917 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.949921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftzq\" (UniqueName: \"kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950215 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.949949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950215 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.949996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950215 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.950134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950215 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.950170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950215 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.950198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:13.950412 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:13.950225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.050915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.050864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.050915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.050917    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.050938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.050963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.050988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftzq\" (UniqueName: \"kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051209 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051511 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051511 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051822 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051822 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.051983 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.051820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.053213 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.053194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.053564 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.053545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.059929 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.059899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftzq\" (UniqueName: \"kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.180680 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.180600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:14.473793 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.473766    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:34:14.476287 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:34:14.476257    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88c57b7_ff2d_4109_9102_f1cb4706bc3e.slice/crio-05b7c2fadae2ff284249b811b2535dc8356a595074d1d3921516121ea2dcd8a3 WatchSource:0}: Error finding container 05b7c2fadae2ff284249b811b2535dc8356a595074d1d3921516121ea2dcd8a3: Status 404 returned error can't find the container with id 05b7c2fadae2ff284249b811b2535dc8356a595074d1d3921516121ea2dcd8a3
Sep 17 21:34:14.821385 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.821349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerStarted","Data":"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"}
Sep 17 21:34:14.821577 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.821403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerStarted","Data":"05b7c2fadae2ff284249b811b2535dc8356a595074d1d3921516121ea2dcd8a3"}
Sep 17 21:34:14.821577 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:14.821481    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:15.830197 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:15.830144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerStarted","Data":"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"}
Sep 17 21:34:17.841779 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:17.841734    2582 generic.go:356] "Generic (PLEG): container finished" podID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerID="e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a" exitCode=0
Sep 17 21:34:17.842300 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:17.841810    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerDied","Data":"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"}
Sep 17 21:34:18.849763 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:18.849721    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerStarted","Data":"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"}
Sep 17 21:34:18.875891 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:18.875831    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podStartSLOduration=3.592638552 podStartE2EDuration="5.875812962s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.28317441s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:18.874230063 +0000 UTC m=+2709.453465913" watchObservedRunningTime="2026-09-17 21:34:18.875812962 +0000 UTC m=+2709.455048849"
Sep 17 21:34:24.181058 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:24.181008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:24.181711 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:24.181121    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:24.182486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:24.182446    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:34:24.195357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:24.195327    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:34:34.181047 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:34.181003    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:34:41.782774 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.782734    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:34:41.786629 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.786589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:41.789134 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.789081    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 21:34:41.790311 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.790181    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 21:34:41.790311 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.790233    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-msjfk\""
Sep 17 21:34:41.790311 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.790244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-q56p2\""
Sep 17 21:34:41.790311 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.790187    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 21:34:41.796942 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.796911    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:34:41.905843 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.905803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:41.906014 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:41.905865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4j8q\" (UniqueName: \"kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.006563 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.006520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.006762 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.006578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4j8q\" (UniqueName: \"kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.009475 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.009448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.015669 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.015634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4j8q\" (UniqueName: \"kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.100634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.100536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:42.382036 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.381953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:34:42.385513 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:34:42.385467    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3646cb0b_ecff_4e56_8a8d_c9adc8461ee0.slice/crio-299a6df9bed8cb1374ed2f8fbd073c028f80e88616e22fb9597cc1c9b8e232a1 WatchSource:0}: Error finding container 299a6df9bed8cb1374ed2f8fbd073c028f80e88616e22fb9597cc1c9b8e232a1: Status 404 returned error can't find the container with id 299a6df9bed8cb1374ed2f8fbd073c028f80e88616e22fb9597cc1c9b8e232a1
Sep 17 21:34:42.970801 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.970762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" event={"ID":"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0","Type":"ContainerStarted","Data":"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"}
Sep 17 21:34:42.971282 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.970850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" event={"ID":"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0","Type":"ContainerStarted","Data":"299a6df9bed8cb1374ed2f8fbd073c028f80e88616e22fb9597cc1c9b8e232a1"}
Sep 17 21:34:42.971282 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:42.970875    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:34:43.009599 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:43.009535    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podStartSLOduration=2.009517915 podStartE2EDuration="2.009517915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:43.007953609 +0000 UTC m=+2733.587189460" watchObservedRunningTime="2026-09-17 21:34:43.009517915 +0000 UTC m=+2733.588753765"
Sep 17 21:34:44.182160 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:44.182080    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:34:54.182032 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:34:54.181982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:04.181463 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:04.181409    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:13.980083 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:13.980049    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:35:14.181350 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:14.181292    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:24.181769 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:24.181551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:34.181770 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:34.181713    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:44.181273 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:44.181227    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:35:54.181358 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:35:54.181310    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:36:04.181878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:04.181824    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8001/health\": dial tcp 10.133.0.43:8001: connect: connection refused"
Sep 17 21:36:14.190909 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:14.190874    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:36:14.207041 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:14.207008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:36:23.152157 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:23.152122    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:36:38.176219 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:38.176172    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main" containerID="cri-o://debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16" gracePeriod=45
Sep 17 21:36:38.787856 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:38.787810    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="llm-d-routing-sidecar" containerID="cri-o://8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815" gracePeriod=45
Sep 17 21:36:39.052961 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.052935    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:36:39.069184 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069160    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sftzq\" (UniqueName: \"kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069271 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069271 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069247    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069339    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069393 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069433 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069422    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069485 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069539 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert\") pod \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\" (UID: \"e88c57b7-ff2d-4109-9102-f1cb4706bc3e\") "
Sep 17 21:36:39.069942 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.069867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.070149 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.070086    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:39.070149 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.070130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home" (OuterVolumeSpecName: "home") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.071661 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.071639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm" (OuterVolumeSpecName: "dshm") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.071804 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.071779    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq" (OuterVolumeSpecName: "kube-api-access-sftzq") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "kube-api-access-sftzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:36:39.072045 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.072021    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:36:39.084064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.084028    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.093991 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.093953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e88c57b7-ff2d-4109-9102-f1cb4706bc3e" (UID: "e88c57b7-ff2d-4109-9102-f1cb4706bc3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.171079 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171035    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171079 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171073    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171079 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171086    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171124    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171138    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171149    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171161    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sftzq\" (UniqueName: \"kubernetes.io/projected/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-kube-api-access-sftzq\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.171340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.171172    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e88c57b7-ff2d-4109-9102-f1cb4706bc3e-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.565615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565580    2582 generic.go:356] "Generic (PLEG): container finished" podID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerID="debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16" exitCode=0
Sep 17 21:36:39.565615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565604    2582 generic.go:356] "Generic (PLEG): container finished" podID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerID="8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815" exitCode=0
Sep 17 21:36:39.566196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565662    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"
Sep 17 21:36:39.566196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerDied","Data":"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"}
Sep 17 21:36:39.566196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerDied","Data":"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"}
Sep 17 21:36:39.566196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl" event={"ID":"e88c57b7-ff2d-4109-9102-f1cb4706bc3e","Type":"ContainerDied","Data":"05b7c2fadae2ff284249b811b2535dc8356a595074d1d3921516121ea2dcd8a3"}
Sep 17 21:36:39.566196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.565864    2582 scope.go:117] "RemoveContainer" containerID="debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"
Sep 17 21:36:39.574813 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.574797    2582 scope.go:117] "RemoveContainer" containerID="e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"
Sep 17 21:36:39.583353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.583334    2582 scope.go:117] "RemoveContainer" containerID="8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"
Sep 17 21:36:39.589294 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.589274    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:36:39.591323 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.591302    2582 scope.go:117] "RemoveContainer" containerID="debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"
Sep 17 21:36:39.591592 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:36:39.591572    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16\": container with ID starting with debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16 not found: ID does not exist" containerID="debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"
Sep 17 21:36:39.591647 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.591604    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"} err="failed to get container status \"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16\": rpc error: code = NotFound desc = could not find container \"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16\": container with ID starting with debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16 not found: ID does not exist"
Sep 17 21:36:39.591647 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.591627    2582 scope.go:117] "RemoveContainer" containerID="e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"
Sep 17 21:36:39.591888 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:36:39.591867    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a\": container with ID starting with e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a not found: ID does not exist" containerID="e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"
Sep 17 21:36:39.591944 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.591892    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"} err="failed to get container status \"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a\": rpc error: code = NotFound desc = could not find container \"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a\": container with ID starting with e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a not found: ID does not exist"
Sep 17 21:36:39.591944 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.591905    2582 scope.go:117] "RemoveContainer" containerID="8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"
Sep 17 21:36:39.592130 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:36:39.592111    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815\": container with ID starting with 8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815 not found: ID does not exist" containerID="8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"
Sep 17 21:36:39.592193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592135    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"} err="failed to get container status \"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815\": rpc error: code = NotFound desc = could not find container \"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815\": container with ID starting with 8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815 not found: ID does not exist"
Sep 17 21:36:39.592193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592147    2582 scope.go:117] "RemoveContainer" containerID="debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"
Sep 17 21:36:39.592518 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592495    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16"} err="failed to get container status \"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16\": rpc error: code = NotFound desc = could not find container \"debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16\": container with ID starting with debe5f6d7a69d47385dc8eeb16329baf8911ea6d90ef6824bd385d5001db7a16 not found: ID does not exist"
Sep 17 21:36:39.592518 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592516    2582 scope.go:117] "RemoveContainer" containerID="e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"
Sep 17 21:36:39.592726 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592705    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a"} err="failed to get container status \"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a\": rpc error: code = NotFound desc = could not find container \"e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a\": container with ID starting with e6098702095e06ec3bdc62189ed3962405f134659d96ba01810857149f77a01a not found: ID does not exist"
Sep 17 21:36:39.592772 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592728    2582 scope.go:117] "RemoveContainer" containerID="8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"
Sep 17 21:36:39.592962 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.592945    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815"} err="failed to get container status \"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815\": rpc error: code = NotFound desc = could not find container \"8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815\": container with ID starting with 8eb3070821239bda6ccf4a8b0330eb4654d0b4878d377bf43e7a883648bd3815 not found: ID does not exist"
Sep 17 21:36:39.596657 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:39.596636    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f8958df66-fg8kl"]
Sep 17 21:36:41.583355 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:41.583320    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" path="/var/lib/kubelet/pods/e88c57b7-ff2d-4109-9102-f1cb4706bc3e/volumes"
Sep 17 21:36:43.768747 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:43.768710    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:36:43.979353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:43.979316    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:53.979475 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:53.979393    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:36:58.791032 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:36:58.790962    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" containerID="cri-o://b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9" gracePeriod=45
Sep 17 21:37:03.979741 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:03.979698    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:37:03.980139 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:03.979806    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:37:13.979186 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:13.979144    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:37:23.937700 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.937661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.937995    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="llm-d-routing-sidecar"
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938006    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="llm-d-routing-sidecar"
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938017    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main"
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938022    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main"
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938036    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="storage-initializer"
Sep 17 21:37:23.938064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938042    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="storage-initializer"
Sep 17 21:37:23.938283 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="llm-d-routing-sidecar"
Sep 17 21:37:23.938283 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.938125    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e88c57b7-ff2d-4109-9102-f1cb4706bc3e" containerName="main"
Sep 17 21:37:23.942030 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.942014    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.948707 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.948683    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:37:23.948857 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.948742    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 21:37:23.948927 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.948910    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 21:37:23.949551 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.949374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 21:37:23.949854 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.949837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 21:37:23.955214 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.955196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-4lxl9\""
Sep 17 21:37:23.958898 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.958879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.958965 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.958916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.958965 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.958949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.959055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.958980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.959055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.959003    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.959055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.959028    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.959189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.959057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkmx\" (UniqueName: \"kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.959189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.959078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:23.978957 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.978930    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:37:23.990199 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:23.990173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:37:24.060268 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060480 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkmx\" (UniqueName: \"kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060741 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060741 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060859 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.060859 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.061012 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.061012 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.060946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.061567 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.061542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.062761 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.062729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.062888 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.062808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.075384 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.075359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkmx\" (UniqueName: \"kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx\") pod \"custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.258798 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.258761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:24.544492 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.544407    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:37:24.546761 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:37:24.546732    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672be3bb_fc75_4de2_87cd_6c66a195466a.slice/crio-60fb91548bc9f043f06bf1cd024e1e28632e0c2f6503464cefd1e3ccdf31eaae WatchSource:0}: Error finding container 60fb91548bc9f043f06bf1cd024e1e28632e0c2f6503464cefd1e3ccdf31eaae: Status 404 returned error can't find the container with id 60fb91548bc9f043f06bf1cd024e1e28632e0c2f6503464cefd1e3ccdf31eaae
Sep 17 21:37:24.771005 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.770969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerStarted","Data":"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"}
Sep 17 21:37:24.771220 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:24.771021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerStarted","Data":"60fb91548bc9f043f06bf1cd024e1e28632e0c2f6503464cefd1e3ccdf31eaae"}
Sep 17 21:37:27.786504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:27.786469    2582 generic.go:356] "Generic (PLEG): container finished" podID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerID="406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674" exitCode=0
Sep 17 21:37:27.786504 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:27.786510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerDied","Data":"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"}
Sep 17 21:37:28.794405 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:28.794372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerStarted","Data":"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"}
Sep 17 21:37:28.830274 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:28.830223    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podStartSLOduration=3.638856342 podStartE2EDuration="5.830206024s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.191349682s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:37:28.826152213 +0000 UTC m=+2899.405388064" watchObservedRunningTime="2026-09-17 21:37:28.830206024 +0000 UTC m=+2899.409441874"
Sep 17 21:37:29.037990 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.037964    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:37:29.096083 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.095997    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs\") pod \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") "
Sep 17 21:37:29.096083 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.096041    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4j8q\" (UniqueName: \"kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q\") pod \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\" (UID: \"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0\") "
Sep 17 21:37:29.098200 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.098170    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" (UID: "3646cb0b-ecff-4e56-8a8d-c9adc8461ee0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:37:29.098377 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.098350    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q" (OuterVolumeSpecName: "kube-api-access-b4j8q") pod "3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" (UID: "3646cb0b-ecff-4e56-8a8d-c9adc8461ee0"). InnerVolumeSpecName "kube-api-access-b4j8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:37:29.197115 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.197052    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:37:29.197115 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.197084    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4j8q\" (UniqueName: \"kubernetes.io/projected/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0-kube-api-access-b4j8q\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:37:29.803239 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.803202    2582 generic.go:356] "Generic (PLEG): container finished" podID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerID="b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9" exitCode=0
Sep 17 21:37:29.803665 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.803253    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"
Sep 17 21:37:29.803665 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.803252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" event={"ID":"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0","Type":"ContainerDied","Data":"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"}
Sep 17 21:37:29.803665 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.803320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g" event={"ID":"3646cb0b-ecff-4e56-8a8d-c9adc8461ee0","Type":"ContainerDied","Data":"299a6df9bed8cb1374ed2f8fbd073c028f80e88616e22fb9597cc1c9b8e232a1"}
Sep 17 21:37:29.803665 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.803361    2582 scope.go:117] "RemoveContainer" containerID="b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"
Sep 17 21:37:29.811970 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.811945    2582 scope.go:117] "RemoveContainer" containerID="b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"
Sep 17 21:37:29.812277 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:37:29.812258    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9\": container with ID starting with b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9 not found: ID does not exist" containerID="b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"
Sep 17 21:37:29.812347 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.812285    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9"} err="failed to get container status \"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9\": rpc error: code = NotFound desc = could not find container \"b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9\": container with ID starting with b7e392a920606814976127657924abc54e5ada1113859fee9cecac4a8510b1f9 not found: ID does not exist"
Sep 17 21:37:29.826481 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.820556    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:37:29.828840 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:29.828812    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-jbv2g"]
Sep 17 21:37:31.584032 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:31.583992    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" path="/var/lib/kubelet/pods/3646cb0b-ecff-4e56-8a8d-c9adc8461ee0/volumes"
Sep 17 21:37:34.259187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:34.259148    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:34.259624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:34.259215    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:37:34.260658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:34.260632    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:37:44.260076 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:44.260023    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:37:54.259760 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:37:54.259719    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:04.259954 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:04.259914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:14.259641 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:14.259597    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:24.259801 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:24.259707    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:34.259526 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:34.259485    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:44.259296 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:44.259244    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:38:54.259552 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:38:54.259498    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:39:04.259353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:04.259308    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:39:11.725137 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:11.725083    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:39:11.730421 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:11.730392    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:39:11.731307 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:11.731285    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:39:11.736415 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:11.736397    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:39:14.259399 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:14.259354    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 21:39:24.273433 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:24.273401    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:39:24.282380 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:24.282352    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:39:33.418220 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:33.418190    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:39:48.441235 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:48.441166    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main" containerID="cri-o://00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737" gracePeriod=45
Sep 17 21:39:49.184977 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.184951    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:39:49.340221 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340179    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjkmx\" (UniqueName: \"kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340233    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340278    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340297    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340338    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340701 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.340701 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.340474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert\") pod \"672be3bb-fc75-4de2-87cd-6c66a195466a\" (UID: \"672be3bb-fc75-4de2-87cd-6c66a195466a\") "
Sep 17 21:39:49.341150 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.341059    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home" (OuterVolumeSpecName: "home") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.341333 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.341060    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:49.341333 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.341189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache" (OuterVolumeSpecName: "model-cache") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.342779 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.342748    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm" (OuterVolumeSpecName: "dshm") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.342903 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.342832    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx" (OuterVolumeSpecName: "kube-api-access-xjkmx") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "kube-api-access-xjkmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:49.343136 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.343087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:49.353044 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.353023    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.359690 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.359667    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "672be3bb-fc75-4de2-87cd-6c66a195466a" (UID: "672be3bb-fc75-4de2-87cd-6c66a195466a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:49.441919 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441886    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.441919 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441914    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/672be3bb-fc75-4de2-87cd-6c66a195466a-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.441919 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441927    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.442388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441935    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/672be3bb-fc75-4de2-87cd-6c66a195466a-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.442388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441944    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkmx\" (UniqueName: \"kubernetes.io/projected/672be3bb-fc75-4de2-87cd-6c66a195466a-kube-api-access-xjkmx\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.442388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441951    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.442388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441962    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.442388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.441969    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/672be3bb-fc75-4de2-87cd-6c66a195466a-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:39:49.484082 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.484048    2582 generic.go:356] "Generic (PLEG): container finished" podID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerID="00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737" exitCode=0
Sep 17 21:39:49.484255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.484119    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"
Sep 17 21:39:49.484255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.484139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerDied","Data":"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"}
Sep 17 21:39:49.484255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.484217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd" event={"ID":"672be3bb-fc75-4de2-87cd-6c66a195466a","Type":"ContainerDied","Data":"60fb91548bc9f043f06bf1cd024e1e28632e0c2f6503464cefd1e3ccdf31eaae"}
Sep 17 21:39:49.484255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.484241    2582 scope.go:117] "RemoveContainer" containerID="00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"
Sep 17 21:39:49.493572 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.493555    2582 scope.go:117] "RemoveContainer" containerID="406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"
Sep 17 21:39:49.501216 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.501194    2582 scope.go:117] "RemoveContainer" containerID="00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"
Sep 17 21:39:49.501492 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:39:49.501473    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737\": container with ID starting with 00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737 not found: ID does not exist" containerID="00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"
Sep 17 21:39:49.501549 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.501500    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737"} err="failed to get container status \"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737\": rpc error: code = NotFound desc = could not find container \"00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737\": container with ID starting with 00a70863812209b54d01b1b62135e13988abb281a9966885fd724e48f7b83737 not found: ID does not exist"
Sep 17 21:39:49.501549 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.501524    2582 scope.go:117] "RemoveContainer" containerID="406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"
Sep 17 21:39:49.501768 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:39:49.501750    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674\": container with ID starting with 406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674 not found: ID does not exist" containerID="406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"
Sep 17 21:39:49.501823 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.501779    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674"} err="failed to get container status \"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674\": rpc error: code = NotFound desc = could not find container \"406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674\": container with ID starting with 406e991fc92f81e8d96a030d3182eb1ee4180aba0b676e352c7377d8ea2aa674 not found: ID does not exist"
Sep 17 21:39:49.507876 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.507854    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:39:49.511985 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.511961    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-769d46547c-6tqmd"]
Sep 17 21:39:49.584709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:39:49.584683    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" path="/var/lib/kubelet/pods/672be3bb-fc75-4de2-87cd-6c66a195466a/volumes"
Sep 17 21:40:00.934297 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934263    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934606    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934617    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934627    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="storage-initializer"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934633    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="storage-initializer"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934646    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934652    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934706    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3646cb0b-ecff-4e56-8a8d-c9adc8461ee0" containerName="main"
Sep 17 21:40:00.934826 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.934716    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="672be3bb-fc75-4de2-87cd-6c66a195466a" containerName="main"
Sep 17 21:40:00.939626 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.939603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:00.942430 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.942404    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 21:40:00.942563 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.942417    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 21:40:00.942563 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.942504    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-vgtnm\""
Sep 17 21:40:00.942563 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.942480    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 21:40:00.942563 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.942527    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:40:00.943212 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.943196    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 21:40:00.952921 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:00.952895    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:40:01.021240 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021240 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021380    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.021634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.021478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmhk\" (UniqueName: \"kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122659 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122659 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.122897 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.122887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmhk\" (UniqueName: \"kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.123249 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.123183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.123308 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.123249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.123369 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.123340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.123527 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.123501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.123527 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.123527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.125040 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.125018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.125459 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.125438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.131423 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.131403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmhk\" (UniqueName: \"kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk\") pod \"lora-single-adapter-test-kserve-b988fc67d-r586t\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.250008 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.249965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:01.534752 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.534722    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:40:01.536682 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:40:01.536646    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc673a91_007e_4d7f_b760_b727e800ffc4.slice/crio-0a551f2bb3cdbf877c471b07e23d079bdbf2255dc3e88e1d3774b51945d195c3 WatchSource:0}: Error finding container 0a551f2bb3cdbf877c471b07e23d079bdbf2255dc3e88e1d3774b51945d195c3: Status 404 returned error can't find the container with id 0a551f2bb3cdbf877c471b07e23d079bdbf2255dc3e88e1d3774b51945d195c3
Sep 17 21:40:01.549423 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:01.549392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerStarted","Data":"0a551f2bb3cdbf877c471b07e23d079bdbf2255dc3e88e1d3774b51945d195c3"}
Sep 17 21:40:02.555705 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:02.555656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerStarted","Data":"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"}
Sep 17 21:40:04.569405 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:04.569371    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerID="4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3" exitCode=0
Sep 17 21:40:04.569898 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:04.569452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerDied","Data":"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"}
Sep 17 21:40:05.575943 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:05.575905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerStarted","Data":"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"}
Sep 17 21:40:05.610127 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:05.610054    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podStartSLOduration=2.821244083 podStartE2EDuration="5.61003404s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.788789957s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:05.607943946 +0000 UTC m=+3056.187179810" watchObservedRunningTime="2026-09-17 21:40:05.61003404 +0000 UTC m=+3056.189269891"
Sep 17 21:40:06.251087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:06.251052    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:11.250584 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:11.250540    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:40:11.252485 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:11.252444    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:40:21.250595 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:21.250526    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:40:31.250656 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:31.250607    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:40:41.251077 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:41.251026    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:40:44.281123 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.281069    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:40:44.284354 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.284334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.287784 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.287759    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 21:40:44.287945 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.287768    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-9tzwm\""
Sep 17 21:40:44.287945 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.287768    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 21:40:44.287945 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.287776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-k9qhp\""
Sep 17 21:40:44.288178 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.287829    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 21:40:44.296707 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.296677    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:40:44.320660 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.320619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grp7f\" (UniqueName: \"kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.320660 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.320659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.421519 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.421477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grp7f\" (UniqueName: \"kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.421703 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.421534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.424358 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.424327    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.431319 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.431288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grp7f\" (UniqueName: \"kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.595595 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.595491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:44.872545 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:44.872486    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:40:44.875955 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:40:44.875923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1075eb5_a829_401d_b0f2_8e38675a0359.slice/crio-7316eb13232d64630ac806722f4c2bba21c27221126d42328a1ab629229d8258 WatchSource:0}: Error finding container 7316eb13232d64630ac806722f4c2bba21c27221126d42328a1ab629229d8258: Status 404 returned error can't find the container with id 7316eb13232d64630ac806722f4c2bba21c27221126d42328a1ab629229d8258
Sep 17 21:40:45.783677 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:45.783627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" event={"ID":"c1075eb5-a829-401d-b0f2-8e38675a0359","Type":"ContainerStarted","Data":"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"}
Sep 17 21:40:45.784131 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:45.783701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" event={"ID":"c1075eb5-a829-401d-b0f2-8e38675a0359","Type":"ContainerStarted","Data":"7316eb13232d64630ac806722f4c2bba21c27221126d42328a1ab629229d8258"}
Sep 17 21:40:45.784131 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:45.783950    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:40:45.802573 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:45.802517    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podStartSLOduration=1.802496704 podStartE2EDuration="1.802496704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:45.801139282 +0000 UTC m=+3096.380375134" watchObservedRunningTime="2026-09-17 21:40:45.802496704 +0000 UTC m=+3096.381732556"
Sep 17 21:40:51.251422 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:40:51.251379    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:41:01.250842 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:01.250788    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:41:11.251361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:11.251315    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 21:41:16.790794 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:16.790756    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:41:21.261219 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:21.261186    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:41:21.269299 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:21.269271    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:41:47.070069 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:47.069965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-b988fc67d-r586t_dc673a91-007e-4d7f-b760-b727e800ffc4/storage-initializer/0.log"
Sep 17 21:41:47.082796 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:47.082755    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-b988fc67d-r586t_dc673a91-007e-4d7f-b760-b727e800ffc4/main/0.log"
Sep 17 21:41:48.407269 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:41:48.407233    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:42:03.429759 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:03.429712    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main" containerID="cri-o://3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88" gracePeriod=45
Sep 17 21:42:04.069565 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.069542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:42:04.159153 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.159049    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerID="3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88" exitCode=0
Sep 17 21:42:04.159153 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.159114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerDied","Data":"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"}
Sep 17 21:42:04.159343 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.159162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t" event={"ID":"dc673a91-007e-4d7f-b760-b727e800ffc4","Type":"ContainerDied","Data":"0a551f2bb3cdbf877c471b07e23d079bdbf2255dc3e88e1d3774b51945d195c3"}
Sep 17 21:42:04.159343 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.159183    2582 scope.go:117] "RemoveContainer" containerID="3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"
Sep 17 21:42:04.159343 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.159120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"
Sep 17 21:42:04.167470 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.167441    2582 scope.go:117] "RemoveContainer" containerID="4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"
Sep 17 21:42:04.175112 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.175080    2582 scope.go:117] "RemoveContainer" containerID="3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"
Sep 17 21:42:04.175355 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:42:04.175334    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88\": container with ID starting with 3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88 not found: ID does not exist" containerID="3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"
Sep 17 21:42:04.175431 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.175368    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88"} err="failed to get container status \"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88\": rpc error: code = NotFound desc = could not find container \"3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88\": container with ID starting with 3641f4866e1b57e0dc71cca24eb4ed3b66955d0466410162c998a2e00bd4ce88 not found: ID does not exist"
Sep 17 21:42:04.175431 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.175395    2582 scope.go:117] "RemoveContainer" containerID="4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"
Sep 17 21:42:04.175654 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:42:04.175637    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3\": container with ID starting with 4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3 not found: ID does not exist" containerID="4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"
Sep 17 21:42:04.175695 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.175658    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3"} err="failed to get container status \"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3\": rpc error: code = NotFound desc = could not find container \"4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3\": container with ID starting with 4749527e5f5042fe8c74a5fb09c65471c5773a0ba36e5f189eb15a36eec3e5b3 not found: ID does not exist"
Sep 17 21:42:04.237015 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.236986    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237021    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237052    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237081    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237139    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqmhk\" (UniqueName: \"kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237193 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237184    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237445 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237445 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237248    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir\") pod \"dc673a91-007e-4d7f-b760-b727e800ffc4\" (UID: \"dc673a91-007e-4d7f-b760-b727e800ffc4\") "
Sep 17 21:42:04.237445 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237343    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:04.237601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:04.237601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237560    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home" (OuterVolumeSpecName: "home") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:04.237601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237573    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.237601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.237590    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dc673a91-007e-4d7f-b760-b727e800ffc4-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.239439 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.239411    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm" (OuterVolumeSpecName: "dshm") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:04.239564 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.239547    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk" (OuterVolumeSpecName: "kube-api-access-gqmhk") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "kube-api-access-gqmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:04.239740 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.239727    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:42:04.253616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.253587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:04.260178 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.260157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc673a91-007e-4d7f-b760-b727e800ffc4" (UID: "dc673a91-007e-4d7f-b760-b727e800ffc4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:04.338544 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338507    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc673a91-007e-4d7f-b760-b727e800ffc4-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.338544 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338535    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.338736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338563    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.338736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338572    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.338736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338582    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc673a91-007e-4d7f-b760-b727e800ffc4-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.338736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.338599    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmhk\" (UniqueName: \"kubernetes.io/projected/dc673a91-007e-4d7f-b760-b727e800ffc4-kube-api-access-gqmhk\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:42:04.484783 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.484752    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:42:04.487597 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:04.487572    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b988fc67d-r586t"]
Sep 17 21:42:05.584273 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:05.584232    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" path="/var/lib/kubelet/pods/dc673a91-007e-4d7f-b760-b727e800ffc4/volumes"
Sep 17 21:42:18.120187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120142    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:42:18.120624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120525    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main"
Sep 17 21:42:18.120624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main"
Sep 17 21:42:18.120624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120555    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="storage-initializer"
Sep 17 21:42:18.120624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120561    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="storage-initializer"
Sep 17 21:42:18.120624 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.120621    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc673a91-007e-4d7f-b760-b727e800ffc4" containerName="main"
Sep 17 21:42:18.122634 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.122618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.124911 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.124885    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 21:42:18.125082 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.124927    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:42:18.125082 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.125060    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 21:42:18.125295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.125173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 21:42:18.125295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.125231    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 21:42:18.125295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.125251    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-wtslr\""
Sep 17 21:42:18.139146 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.139117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:42:18.155791 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.155978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.155978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.155978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.155978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9j8\" (UniqueName: \"kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.156207 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.155998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.156207 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.156031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.156207 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.156057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256484 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9j8\" (UniqueName: \"kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.256704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.256697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.257112 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.257064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.257255 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.257230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.257342 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.257246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.257417 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.257361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.257417 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.257393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.259172 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.259140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.259416 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.259400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.264314 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.264292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9j8\" (UniqueName: \"kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8\") pod \"lora-multiple-adapters-test-kserve-7b4545875c-xh9bf\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.433174 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.433053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:18.718434 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:18.718394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:42:18.721254 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:42:18.721223    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5412be9c_3327_433d_830b_ae9f36110e0e.slice/crio-fb7758a4275ec9cf4787c0273b8bef69a13857b0ee52139d4241529648b8d7a1 WatchSource:0}: Error finding container fb7758a4275ec9cf4787c0273b8bef69a13857b0ee52139d4241529648b8d7a1: Status 404 returned error can't find the container with id fb7758a4275ec9cf4787c0273b8bef69a13857b0ee52139d4241529648b8d7a1
Sep 17 21:42:19.232534 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:19.232499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerStarted","Data":"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"}
Sep 17 21:42:19.233031 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:19.232566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerStarted","Data":"fb7758a4275ec9cf4787c0273b8bef69a13857b0ee52139d4241529648b8d7a1"}
Sep 17 21:42:22.250415 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:22.250376    2582 generic.go:356] "Generic (PLEG): container finished" podID="5412be9c-3327-433d-830b-ae9f36110e0e" containerID="0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f" exitCode=0
Sep 17 21:42:22.250878 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:22.250432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerDied","Data":"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"}
Sep 17 21:42:23.256927 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:23.256891    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerStarted","Data":"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"}
Sep 17 21:42:23.279780 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:23.279717    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podStartSLOduration=2.352309415 podStartE2EDuration="5.279699812s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.927390397s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:23.27844691 +0000 UTC m=+3193.857682776" watchObservedRunningTime="2026-09-17 21:42:23.279699812 +0000 UTC m=+3193.858935663"
Sep 17 21:42:23.434137 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:23.434081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:28.433555 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:28.433511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:42:28.435254 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:28.435226    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:42:38.433892 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:38.433847    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:42:48.433729 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:48.433687    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:42:53.288198 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:53.288064    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:42:56.791295 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:56.791252    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:42:58.433641 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:42:58.433602    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:43:06.791555 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:06.791508    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:43:08.308037 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:08.307972    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" containerID="cri-o://913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9" gracePeriod=45
Sep 17 21:43:08.434075 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:08.434025    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:43:16.791018 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:16.790975    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:43:16.791581 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:16.791080    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:43:18.434489 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:18.434446    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:43:26.791517 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:26.791479    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:43:28.433609 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:28.433563    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 21:43:36.791682 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:36.791642    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:43:38.444195 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.444162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:43:38.451975 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.451947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:43:38.564053 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.563990    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:43:38.614800 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.614765    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerID="913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9" exitCode=0
Sep 17 21:43:38.614977 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.614815    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"
Sep 17 21:43:38.614977 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.614845    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" event={"ID":"c1075eb5-a829-401d-b0f2-8e38675a0359","Type":"ContainerDied","Data":"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"}
Sep 17 21:43:38.614977 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.614892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s" event={"ID":"c1075eb5-a829-401d-b0f2-8e38675a0359","Type":"ContainerDied","Data":"7316eb13232d64630ac806722f4c2bba21c27221126d42328a1ab629229d8258"}
Sep 17 21:43:38.614977 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.614914    2582 scope.go:117] "RemoveContainer" containerID="913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"
Sep 17 21:43:38.623675 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.623657    2582 scope.go:117] "RemoveContainer" containerID="913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"
Sep 17 21:43:38.623937 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:43:38.623921    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9\": container with ID starting with 913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9 not found: ID does not exist" containerID="913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"
Sep 17 21:43:38.623980 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.623946    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9"} err="failed to get container status \"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9\": rpc error: code = NotFound desc = could not find container \"913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9\": container with ID starting with 913f1a4dfa06d4b773c1bf14c7efbcda918729f3c3bad2f92608924fd8e41fa9 not found: ID does not exist"
Sep 17 21:43:38.666696 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.666667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs\") pod \"c1075eb5-a829-401d-b0f2-8e38675a0359\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") "
Sep 17 21:43:38.666812 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.666757    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grp7f\" (UniqueName: \"kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f\") pod \"c1075eb5-a829-401d-b0f2-8e38675a0359\" (UID: \"c1075eb5-a829-401d-b0f2-8e38675a0359\") "
Sep 17 21:43:38.668956 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.668927    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1075eb5-a829-401d-b0f2-8e38675a0359" (UID: "c1075eb5-a829-401d-b0f2-8e38675a0359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:43:38.669044 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.669007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f" (OuterVolumeSpecName: "kube-api-access-grp7f") pod "c1075eb5-a829-401d-b0f2-8e38675a0359" (UID: "c1075eb5-a829-401d-b0f2-8e38675a0359"). InnerVolumeSpecName "kube-api-access-grp7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:38.767722 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.767690    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grp7f\" (UniqueName: \"kubernetes.io/projected/c1075eb5-a829-401d-b0f2-8e38675a0359-kube-api-access-grp7f\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:43:38.767722 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.767719    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1075eb5-a829-401d-b0f2-8e38675a0359-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:43:38.935522 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.935490    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:43:38.943419 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:38.943392    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zqm4s"]
Sep 17 21:43:39.584495 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:43:39.584463    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" path="/var/lib/kubelet/pods/c1075eb5-a829-401d-b0f2-8e38675a0359/volumes"
Sep 17 21:44:05.390275 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:05.390238    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7b4545875c-xh9bf_5412be9c-3327-433d-830b-ae9f36110e0e/storage-initializer/0.log"
Sep 17 21:44:05.422195 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:05.422166    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-7b4545875c-xh9bf_5412be9c-3327-433d-830b-ae9f36110e0e/main/0.log"
Sep 17 21:44:06.677994 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:06.677959    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:44:11.760532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:11.760503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:44:11.765688 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:11.765666    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:44:11.768744 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:11.768722    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:44:11.773118 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:11.773075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:44:21.699893 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:21.699852    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main" containerID="cri-o://0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1" gracePeriod=45
Sep 17 21:44:22.347251 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.347227    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:44:22.436040 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.435963    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm9j8\" (UniqueName: \"kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436040 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436046    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436126    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436187    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location\") pod \"5412be9c-3327-433d-830b-ae9f36110e0e\" (UID: \"5412be9c-3327-433d-830b-ae9f36110e0e\") "
Sep 17 21:44:22.436614 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home" (OuterVolumeSpecName: "home") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:22.436614 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436565    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:22.436792 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.436740    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:44:22.438466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.438432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8" (OuterVolumeSpecName: "kube-api-access-gm9j8") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "kube-api-access-gm9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:44:22.438466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.438445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:44:22.438643 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.438527    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm" (OuterVolumeSpecName: "dshm") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:22.452629 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.452598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:22.460938 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.460917    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5412be9c-3327-433d-830b-ae9f36110e0e" (UID: "5412be9c-3327-433d-830b-ae9f36110e0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:22.537736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537707    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537736 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537731    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5412be9c-3327-433d-830b-ae9f36110e0e-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537742    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537752    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537761    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9j8\" (UniqueName: \"kubernetes.io/projected/5412be9c-3327-433d-830b-ae9f36110e0e-kube-api-access-gm9j8\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537768    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537777    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5412be9c-3327-433d-830b-ae9f36110e0e-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.537906 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.537784    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5412be9c-3327-433d-830b-ae9f36110e0e-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:44:22.830837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.830803    2582 generic.go:356] "Generic (PLEG): container finished" podID="5412be9c-3327-433d-830b-ae9f36110e0e" containerID="0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1" exitCode=0
Sep 17 21:44:22.831366 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.830849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerDied","Data":"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"}
Sep 17 21:44:22.831366 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.830861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"
Sep 17 21:44:22.831366 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.830901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf" event={"ID":"5412be9c-3327-433d-830b-ae9f36110e0e","Type":"ContainerDied","Data":"fb7758a4275ec9cf4787c0273b8bef69a13857b0ee52139d4241529648b8d7a1"}
Sep 17 21:44:22.831366 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.830933    2582 scope.go:117] "RemoveContainer" containerID="0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"
Sep 17 21:44:22.840086 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.840067    2582 scope.go:117] "RemoveContainer" containerID="0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"
Sep 17 21:44:22.848128 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.848104    2582 scope.go:117] "RemoveContainer" containerID="0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"
Sep 17 21:44:22.848374 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:44:22.848357    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1\": container with ID starting with 0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1 not found: ID does not exist" containerID="0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"
Sep 17 21:44:22.848451 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.848385    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1"} err="failed to get container status \"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1\": rpc error: code = NotFound desc = could not find container \"0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1\": container with ID starting with 0d42ceae421308dd93a900ce91bb73d8da5dd5e35e59d9044ce55258d8be92d1 not found: ID does not exist"
Sep 17 21:44:22.848451 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.848409    2582 scope.go:117] "RemoveContainer" containerID="0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"
Sep 17 21:44:22.848683 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:44:22.848664    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f\": container with ID starting with 0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f not found: ID does not exist" containerID="0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"
Sep 17 21:44:22.848725 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.848689    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f"} err="failed to get container status \"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f\": rpc error: code = NotFound desc = could not find container \"0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f\": container with ID starting with 0216382ab5e908cfc054662877803b8cfadd145b556358e4836383a83597a28f not found: ID does not exist"
Sep 17 21:44:22.852412 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.852391    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:44:22.858276 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:22.858250    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-7b4545875c-xh9bf"]
Sep 17 21:44:23.584220 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:44:23.584186    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" path="/var/lib/kubelet/pods/5412be9c-3327-433d-830b-ae9f36110e0e/volumes"
Sep 17 21:46:30.603120 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603438    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603453    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603477    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="storage-initializer"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603483    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="storage-initializer"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603492    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603499    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603545    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5412be9c-3327-433d-830b-ae9f36110e0e" containerName="main"
Sep 17 21:46:30.603559 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.603555    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1075eb5-a829-401d-b0f2-8e38675a0359" containerName="main"
Sep 17 21:46:30.606298 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.606281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.610726 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.610702    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-g7zk5\""
Sep 17 21:46:30.610945 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.610733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-2dr27\""
Sep 17 21:46:30.611055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.610737    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 21:46:30.611055 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.610785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 21:46:30.611777 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.611756    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 21:46:30.624219 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.624193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:46:30.680270 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.680240    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:46:30.683648 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.683629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.688023 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.688006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xcc7f\""
Sep 17 21:46:30.688240 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.688222    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:46:30.688572 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.688558    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 21:46:30.700463 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.700441    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:46:30.717424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.717147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.717424 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.717268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ns4\" (UniqueName: \"kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.818040 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ns4\" (UniqueName: \"kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.818225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818225 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818473 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818473 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fccpw\" (UniqueName: \"kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.818473 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.818473 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.818437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.820912 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.820891    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.826955 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.826936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ns4\" (UniqueName: \"kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.917187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.917085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:30.919506 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919620 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919807 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919807 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fccpw\" (UniqueName: \"kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.919807 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.920148 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.920148 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.919957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.920148 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.920122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.920623 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.920259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.920623 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.920463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.921922 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.921897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.922237 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.922214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.929507 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.929481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fccpw\" (UniqueName: \"kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:30.993633 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:30.993605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:46:31.198705 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.198618    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:46:31.202007 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:46:31.201976    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec5a22c_2c9e_4758_b386_76eee33e065c.slice/crio-0f02faca0d8eb7ebb70393c96f693542df4837ec4fcfdbe7a55b65604cdbb536 WatchSource:0}: Error finding container 0f02faca0d8eb7ebb70393c96f693542df4837ec4fcfdbe7a55b65604cdbb536: Status 404 returned error can't find the container with id 0f02faca0d8eb7ebb70393c96f693542df4837ec4fcfdbe7a55b65604cdbb536
Sep 17 21:46:31.281566 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.281541    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:46:31.283756 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:46:31.283726    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df8738a_7579_417b_9ce2_05e0e417dc16.slice/crio-1576c7b43813277319b94545331eadab4cfdf5a318fd24007436266663e9c1b0 WatchSource:0}: Error finding container 1576c7b43813277319b94545331eadab4cfdf5a318fd24007436266663e9c1b0: Status 404 returned error can't find the container with id 1576c7b43813277319b94545331eadab4cfdf5a318fd24007436266663e9c1b0
Sep 17 21:46:31.401318 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.401271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerStarted","Data":"4e6b18ceecbc2e6407e69a1170f334531829a551edbdc91a29e90edba7ac8262"}
Sep 17 21:46:31.401530 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.401341    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerStarted","Data":"1576c7b43813277319b94545331eadab4cfdf5a318fd24007436266663e9c1b0"}
Sep 17 21:46:31.402931 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.402905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" event={"ID":"9ec5a22c-2c9e-4758-b386-76eee33e065c","Type":"ContainerStarted","Data":"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"}
Sep 17 21:46:31.403047 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.402945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" event={"ID":"9ec5a22c-2c9e-4758-b386-76eee33e065c","Type":"ContainerStarted","Data":"0f02faca0d8eb7ebb70393c96f693542df4837ec4fcfdbe7a55b65604cdbb536"}
Sep 17 21:46:31.403047 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:31.402968    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:46:34.418292 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:34.418256    2582 generic.go:356] "Generic (PLEG): container finished" podID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerID="4e6b18ceecbc2e6407e69a1170f334531829a551edbdc91a29e90edba7ac8262" exitCode=0
Sep 17 21:46:34.418670 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:34.418331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerDied","Data":"4e6b18ceecbc2e6407e69a1170f334531829a551edbdc91a29e90edba7ac8262"}
Sep 17 21:46:34.444643 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:34.444593    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podStartSLOduration=4.444576048 podStartE2EDuration="4.444576048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:46:31.477246698 +0000 UTC m=+3442.056482560" watchObservedRunningTime="2026-09-17 21:46:34.444576048 +0000 UTC m=+3445.023811898"
Sep 17 21:46:35.428680 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:35.428647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerStarted","Data":"2cff661b95346310600de3a7886ae181c6b55c92919835b75a5ef41c7338a74f"}
Sep 17 21:46:35.460824 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:46:35.460762    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.253270161 podStartE2EDuration="5.460737842s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.207467681s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:46:35.458436021 +0000 UTC m=+3446.037671876" watchObservedRunningTime="2026-09-17 21:46:35.460737842 +0000 UTC m=+3446.039973695"
Sep 17 21:47:02.409928 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:47:02.409895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:48:50.656981 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:48:50.656945    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:48:52.409886 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:48:52.409847    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:48:52.578192 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:48:52.578162    2582 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xcc7f\" not found"
Sep 17 21:48:52.620703 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:52.620672    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:52.620845 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:52.620736    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:48:53.120720263 +0000 UTC m=+3583.699956093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:53.123625 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:53.123595    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:53.123797 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:53.123659    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:48:54.12364564 +0000 UTC m=+3584.702881469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:54.131348 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:54.131307    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:54.131839 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:54.131398    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:48:56.131379586 +0000 UTC m=+3586.710615416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:56.150574 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:56.150542    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:48:56.150972 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:48:56.150619    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:49:00.150600613 +0000 UTC m=+3590.729836460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:49:00.185943 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:49:00.185910    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:49:00.186372 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:49:00.185992    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:49:08.185975192 +0000 UTC m=+3598.765211026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:49:02.409514 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:02.409475    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:49:05.676275 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:05.676235    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" containerID="cri-o://b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0" gracePeriod=45
Sep 17 21:49:07.474353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:07.474318    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:49:08.255017 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:49:08.254975    2582 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:49:08.255227 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:49:08.255052    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs podName:4df8738a-7579-417b-9ce2-05e0e417dc16 nodeName:}" failed. No retries permitted until 2026-09-17 21:49:24.255037739 +0000 UTC m=+3614.834273568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 17 21:49:11.789328 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:11.789301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:49:11.793952 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:11.793928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:49:11.799568 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:11.799549    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:49:11.804216 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:11.804196    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:49:12.410025 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:12.409992    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:49:12.410216 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:12.410126    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:49:22.410066 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:22.410030    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:49:22.496110 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:22.496063    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="main" containerID="cri-o://2cff661b95346310600de3a7886ae181c6b55c92919835b75a5ef41c7338a74f" gracePeriod=45
Sep 17 21:49:23.223289 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.223256    2582 generic.go:356] "Generic (PLEG): container finished" podID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerID="2cff661b95346310600de3a7886ae181c6b55c92919835b75a5ef41c7338a74f" exitCode=0
Sep 17 21:49:23.223435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.223331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerDied","Data":"2cff661b95346310600de3a7886ae181c6b55c92919835b75a5ef41c7338a74f"}
Sep 17 21:49:23.248757 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.248736    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:49:23.281476 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281583 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281477    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281583 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fccpw\" (UniqueName: \"kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281583 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281559    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281594    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281626    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.281755 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281651    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.282488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281926    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:49:23.282488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.281984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache\") pod \"4df8738a-7579-417b-9ce2-05e0e417dc16\" (UID: \"4df8738a-7579-417b-9ce2-05e0e417dc16\") "
Sep 17 21:49:23.282488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.282234    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home" (OuterVolumeSpecName: "home") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:23.282488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.282278    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4df8738a-7579-417b-9ce2-05e0e417dc16-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.282488 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.282457    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache" (OuterVolumeSpecName: "model-cache") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:23.284455 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.284370    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw" (OuterVolumeSpecName: "kube-api-access-fccpw") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "kube-api-access-fccpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:23.284455 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.284440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm" (OuterVolumeSpecName: "dshm") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:23.284593 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.284499    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:49:23.293464 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.293443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:23.304001 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.303979    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4df8738a-7579-417b-9ce2-05e0e417dc16" (UID: "4df8738a-7579-417b-9ce2-05e0e417dc16"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:49:23.383388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383367    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383388 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383386    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fccpw\" (UniqueName: \"kubernetes.io/projected/4df8738a-7579-417b-9ce2-05e0e417dc16-kube-api-access-fccpw\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383395    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4df8738a-7579-417b-9ce2-05e0e417dc16-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383404    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383411    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383419    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:23.383520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:23.383426    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4df8738a-7579-417b-9ce2-05e0e417dc16-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:24.229503 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.229465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"4df8738a-7579-417b-9ce2-05e0e417dc16","Type":"ContainerDied","Data":"1576c7b43813277319b94545331eadab4cfdf5a318fd24007436266663e9c1b0"}
Sep 17 21:49:24.229907 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.229532    2582 scope.go:117] "RemoveContainer" containerID="2cff661b95346310600de3a7886ae181c6b55c92919835b75a5ef41c7338a74f"
Sep 17 21:49:24.229907 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.229480    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 21:49:24.237971 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.237954    2582 scope.go:117] "RemoveContainer" containerID="4e6b18ceecbc2e6407e69a1170f334531829a551edbdc91a29e90edba7ac8262"
Sep 17 21:49:24.249482 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.249458    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:49:24.254415 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:24.254394    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 21:49:25.583564 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:25.583530    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" path="/var/lib/kubelet/pods/4df8738a-7579-417b-9ce2-05e0e417dc16/volumes"
Sep 17 21:49:32.409472 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:32.409431    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:49:35.922350 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:35.922320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:49:35.984966 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:35.984895    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74ns4\" (UniqueName: \"kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4\") pod \"9ec5a22c-2c9e-4758-b386-76eee33e065c\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") "
Sep 17 21:49:35.984966 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:35.984938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs\") pod \"9ec5a22c-2c9e-4758-b386-76eee33e065c\" (UID: \"9ec5a22c-2c9e-4758-b386-76eee33e065c\") "
Sep 17 21:49:35.986923 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:35.986897    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4" (OuterVolumeSpecName: "kube-api-access-74ns4") pod "9ec5a22c-2c9e-4758-b386-76eee33e065c" (UID: "9ec5a22c-2c9e-4758-b386-76eee33e065c"). InnerVolumeSpecName "kube-api-access-74ns4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:49:35.987022 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:35.987002    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ec5a22c-2c9e-4758-b386-76eee33e065c" (UID: "9ec5a22c-2c9e-4758-b386-76eee33e065c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:49:36.085839 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.085814    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ns4\" (UniqueName: \"kubernetes.io/projected/9ec5a22c-2c9e-4758-b386-76eee33e065c-kube-api-access-74ns4\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:36.085942 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.085864    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ec5a22c-2c9e-4758-b386-76eee33e065c-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:49:36.286542 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.286509    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerID="b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0" exitCode=0
Sep 17 21:49:36.286709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.286547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" event={"ID":"9ec5a22c-2c9e-4758-b386-76eee33e065c","Type":"ContainerDied","Data":"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"}
Sep 17 21:49:36.286709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.286553    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"
Sep 17 21:49:36.286709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.286601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj" event={"ID":"9ec5a22c-2c9e-4758-b386-76eee33e065c","Type":"ContainerDied","Data":"0f02faca0d8eb7ebb70393c96f693542df4837ec4fcfdbe7a55b65604cdbb536"}
Sep 17 21:49:36.286709 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.286630    2582 scope.go:117] "RemoveContainer" containerID="b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"
Sep 17 21:49:36.295915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.295898    2582 scope.go:117] "RemoveContainer" containerID="b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"
Sep 17 21:49:36.296257 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:49:36.296237    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0\": container with ID starting with b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0 not found: ID does not exist" containerID="b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"
Sep 17 21:49:36.296329 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.296265    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0"} err="failed to get container status \"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0\": rpc error: code = NotFound desc = could not find container \"b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0\": container with ID starting with b1be7e47a1dca6563d6a4286985081bbd0600bd2e5378f032f45f01a2407e5b0 not found: ID does not exist"
Sep 17 21:49:36.308884 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.308861    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:49:36.313588 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:36.313565    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche4qcfj"]
Sep 17 21:49:37.584304 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:37.584267    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" path="/var/lib/kubelet/pods/9ec5a22c-2c9e-4758-b386-76eee33e065c/volumes"
Sep 17 21:49:46.541975 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.541945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542279    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="storage-initializer"
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542291    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="storage-initializer"
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542306    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="main"
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542314    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="main"
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542321    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main"
Sep 17 21:49:46.542351 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542326    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main"
Sep 17 21:49:46.542548 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542381    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ec5a22c-2c9e-4758-b386-76eee33e065c" containerName="main"
Sep 17 21:49:46.542548 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.542387    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df8738a-7579-417b-9ce2-05e0e417dc16" containerName="main"
Sep 17 21:49:46.545221 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.545197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.549264 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.549244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 21:49:46.549384 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.549315    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 21:49:46.549457 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.549379    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 21:49:46.549457 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.549446    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-r2nn2\""
Sep 17 21:49:46.549555 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.549509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-9hp86\""
Sep 17 21:49:46.555466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.555444    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:49:46.672919 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.672887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dl5\" (UniqueName: \"kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.673060 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.672927    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.773626 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.773581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dl5\" (UniqueName: \"kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.773626 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.773631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.776177 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.776156    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.784014 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.783993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dl5\" (UniqueName: \"kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:46.856525 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:46.856445    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:47.138961 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:47.138938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:49:47.140834 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:49:47.140807    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5829c177_a489_4d29_822d_ebad17e4b88e.slice/crio-7144066820e07a9e259165efc4a15a7d5e9f9283831a70496abee707f7762244 WatchSource:0}: Error finding container 7144066820e07a9e259165efc4a15a7d5e9f9283831a70496abee707f7762244: Status 404 returned error can't find the container with id 7144066820e07a9e259165efc4a15a7d5e9f9283831a70496abee707f7762244
Sep 17 21:49:47.340616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:47.340587    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" event={"ID":"5829c177-a489-4d29-822d-ebad17e4b88e","Type":"ContainerStarted","Data":"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"}
Sep 17 21:49:47.340771 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:47.340634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" event={"ID":"5829c177-a489-4d29-822d-ebad17e4b88e","Type":"ContainerStarted","Data":"7144066820e07a9e259165efc4a15a7d5e9f9283831a70496abee707f7762244"}
Sep 17 21:49:47.340771 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:47.340714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:49:47.366961 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:49:47.366909    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podStartSLOduration=1.366896071 podStartE2EDuration="1.366896071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:49:47.366854371 +0000 UTC m=+3637.946090239" watchObservedRunningTime="2026-09-17 21:49:47.366896071 +0000 UTC m=+3637.946131922"
Sep 17 21:50:18.347187 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:18.347153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:50:19.846936 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:19.846902    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:50:28.347547 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:28.347462    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:50:34.866050 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:34.865990    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main" containerID="cri-o://d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f" gracePeriod=45
Sep 17 21:50:38.346884 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:38.346841    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:50:48.347163 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:48.347118    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:50:48.347648 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:48.347225    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:50:58.347254 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:50:58.347215    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 21:51:05.122375 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.122308    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:51:05.147257 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.147231    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs\") pod \"5829c177-a489-4d29-822d-ebad17e4b88e\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") "
Sep 17 21:51:05.147420 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.147281    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8dl5\" (UniqueName: \"kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5\") pod \"5829c177-a489-4d29-822d-ebad17e4b88e\" (UID: \"5829c177-a489-4d29-822d-ebad17e4b88e\") "
Sep 17 21:51:05.149466 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.149437    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5829c177-a489-4d29-822d-ebad17e4b88e" (UID: "5829c177-a489-4d29-822d-ebad17e4b88e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:51:05.149596 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.149574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5" (OuterVolumeSpecName: "kube-api-access-d8dl5") pod "5829c177-a489-4d29-822d-ebad17e4b88e" (UID: "5829c177-a489-4d29-822d-ebad17e4b88e"). InnerVolumeSpecName "kube-api-access-d8dl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:51:05.247808 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.247765    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5829c177-a489-4d29-822d-ebad17e4b88e-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:05.247808 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.247802    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dl5\" (UniqueName: \"kubernetes.io/projected/5829c177-a489-4d29-822d-ebad17e4b88e-kube-api-access-d8dl5\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:05.690434 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.690397    2582 generic.go:356] "Generic (PLEG): container finished" podID="5829c177-a489-4d29-822d-ebad17e4b88e" containerID="d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f" exitCode=0
Sep 17 21:51:05.690615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.690446    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"
Sep 17 21:51:05.690615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.690481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" event={"ID":"5829c177-a489-4d29-822d-ebad17e4b88e","Type":"ContainerDied","Data":"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"}
Sep 17 21:51:05.690615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.690544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh" event={"ID":"5829c177-a489-4d29-822d-ebad17e4b88e","Type":"ContainerDied","Data":"7144066820e07a9e259165efc4a15a7d5e9f9283831a70496abee707f7762244"}
Sep 17 21:51:05.690615 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.690569    2582 scope.go:117] "RemoveContainer" containerID="d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"
Sep 17 21:51:05.699405 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.699381    2582 scope.go:117] "RemoveContainer" containerID="d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"
Sep 17 21:51:05.699672 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:51:05.699650    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f\": container with ID starting with d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f not found: ID does not exist" containerID="d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"
Sep 17 21:51:05.699747 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.699684    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f"} err="failed to get container status \"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f\": rpc error: code = NotFound desc = could not find container \"d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f\": container with ID starting with d7b573536beb53b7adab4a5940a4f0e41bf3caa14f9d47543f014c9e4f7d270f not found: ID does not exist"
Sep 17 21:51:05.707442 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.707413    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:51:05.709062 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:05.709041    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbbrhh"]
Sep 17 21:51:07.584272 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:07.584240    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" path="/var/lib/kubelet/pods/5829c177-a489-4d29-822d-ebad17e4b88e/volumes"
Sep 17 21:51:15.605561 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.605456    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:15.610916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.605781    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main"
Sep 17 21:51:15.610916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.605794    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main"
Sep 17 21:51:15.610916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.605848    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5829c177-a489-4d29-822d-ebad17e4b88e" containerName="main"
Sep 17 21:51:15.610916 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.609933    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.612533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.612515    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 21:51:15.613497 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.613467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 21:51:15.613582 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.613507    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-5956t\""
Sep 17 21:51:15.613720 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.613707    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 21:51:15.625115 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.620848    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:15.636051 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.636196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.636196 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.636327 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.636327 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.636434 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.636325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkhk\" (UniqueName: \"kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737005 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.736973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737005 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737260 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkhk\" (UniqueName: \"kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737260 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737260 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737260 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737493 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737556 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737518    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.737606 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.737527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.739240 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.739219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.739509 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.739492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.745567 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.745537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkhk\" (UniqueName: \"kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:15.921683 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:15.921599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:16.193529 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:16.193496    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:16.196052 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:51:16.196023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61adf6b_8e17_40ed_ae12_004ccafda4cb.slice/crio-af27c7f108e8ed57c4e5a8fd012cf499e64971ae8cf8027fc5c1fc64308b2bdd WatchSource:0}: Error finding container af27c7f108e8ed57c4e5a8fd012cf499e64971ae8cf8027fc5c1fc64308b2bdd: Status 404 returned error can't find the container with id af27c7f108e8ed57c4e5a8fd012cf499e64971ae8cf8027fc5c1fc64308b2bdd
Sep 17 21:51:16.740899 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:16.740867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" event={"ID":"f61adf6b-8e17-40ed-ae12-004ccafda4cb","Type":"ContainerStarted","Data":"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"}
Sep 17 21:51:16.741339 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:16.740929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" event={"ID":"f61adf6b-8e17-40ed-ae12-004ccafda4cb","Type":"ContainerStarted","Data":"af27c7f108e8ed57c4e5a8fd012cf499e64971ae8cf8027fc5c1fc64308b2bdd"}
Sep 17 21:51:16.759667 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:16.759603    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" podStartSLOduration=1.7595868879999998 podStartE2EDuration="1.759586888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:51:16.759002742 +0000 UTC m=+3727.338238593" watchObservedRunningTime="2026-09-17 21:51:16.759586888 +0000 UTC m=+3727.338822739"
Sep 17 21:51:16.922697 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:16.922664    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:25.922017 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:25.921984    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:25.934486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:25.934459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:26.798334 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:26.798302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:49.043539 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.043507    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:49.052718 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:51:49.052676    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:51:49.052718 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:51:49.052718 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:51:49.052718 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7" cmd=["/bin/sleep","15"]
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.052737    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" message="" err=<
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:51:49.052749    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerName="main" containerID="cri-o://3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"
Sep 17 21:51:49.053053 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.052795    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerName="main" containerID="cri-o://3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7" gracePeriod=60
Sep 17 21:51:49.394723 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.394693    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:49.530064 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530261 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530127    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530261 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530155    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530261 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llkhk\" (UniqueName: \"kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530261 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530222    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530491 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530279    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs\") pod \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\" (UID: \"f61adf6b-8e17-40ed-ae12-004ccafda4cb\") "
Sep 17 21:51:49.530491 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530353    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:51:49.530491 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home" (OuterVolumeSpecName: "home") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:51:49.530629 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530531    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:51:49.530629 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530548    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.530629 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.530568    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.532533 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.532508    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:51:49.532664 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.532534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm" (OuterVolumeSpecName: "dshm") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:51:49.532664 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.532636    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk" (OuterVolumeSpecName: "kube-api-access-llkhk") pod "f61adf6b-8e17-40ed-ae12-004ccafda4cb" (UID: "f61adf6b-8e17-40ed-ae12-004ccafda4cb"). InnerVolumeSpecName "kube-api-access-llkhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:51:49.631524 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.631495    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.631524 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.631521    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkhk\" (UniqueName: \"kubernetes.io/projected/f61adf6b-8e17-40ed-ae12-004ccafda4cb-kube-api-access-llkhk\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.631524 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.631534    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.631756 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.631545    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61adf6b-8e17-40ed-ae12-004ccafda4cb-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:51:49.891537 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.891452    2582 generic.go:356] "Generic (PLEG): container finished" podID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerID="3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7" exitCode=0
Sep 17 21:51:49.891537 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.891504    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"
Sep 17 21:51:49.891716 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.891538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" event={"ID":"f61adf6b-8e17-40ed-ae12-004ccafda4cb","Type":"ContainerDied","Data":"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"}
Sep 17 21:51:49.891716 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.891594    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c" event={"ID":"f61adf6b-8e17-40ed-ae12-004ccafda4cb","Type":"ContainerDied","Data":"af27c7f108e8ed57c4e5a8fd012cf499e64971ae8cf8027fc5c1fc64308b2bdd"}
Sep 17 21:51:49.891716 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.891624    2582 scope.go:117] "RemoveContainer" containerID="3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"
Sep 17 21:51:49.900797 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.900771    2582 scope.go:117] "RemoveContainer" containerID="3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"
Sep 17 21:51:49.901052 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:51:49.901031    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7\": container with ID starting with 3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7 not found: ID does not exist" containerID="3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"
Sep 17 21:51:49.901120 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.901062    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7"} err="failed to get container status \"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7\": rpc error: code = NotFound desc = could not find container \"3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7\": container with ID starting with 3a0d509aa224fb46ad9e8468165d1d1141a8a420d7244afc8b40cbbe9bef0aa7 not found: ID does not exist"
Sep 17 21:51:49.907374 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.907353    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:49.910751 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:49.910730    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fklt2c"]
Sep 17 21:51:51.584494 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:51:51.584457    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" path="/var/lib/kubelet/pods/f61adf6b-8e17-40ed-ae12-004ccafda4cb/volumes"
Sep 17 21:52:43.706186 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.706150    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:52:43.706789 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.706472    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerName="main"
Sep 17 21:52:43.706789 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.706483    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerName="main"
Sep 17 21:52:43.706789 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.706537    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f61adf6b-8e17-40ed-ae12-004ccafda4cb" containerName="main"
Sep 17 21:52:43.709479 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.709456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.712398 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.712373    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-2xztj\""
Sep 17 21:52:43.712521 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.712373    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 21:52:43.712521 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.712374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 21:52:43.712521 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.712408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 21:52:43.720472 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.720451    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:52:43.885451 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885416    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.885658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjfw\" (UniqueName: \"kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.885658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.885658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.885658 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.885831 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.885678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986144 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986125    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjfw\" (UniqueName: \"kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986357 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986540 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986514    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986580 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.986627 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.986583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.988498 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.988480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.988638 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.988622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:43.994473 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:43.994453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjfw\" (UniqueName: \"kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:44.019451 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:44.019423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:44.296087 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:44.296055    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:52:44.297746 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:52:44.297715    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604fa74d_54bd_42ac_aae4_04f3e09e316c.slice/crio-80cf63c8938396fe82b71a82909ef11976a57c0461e7534ea813cdedf267e4e1 WatchSource:0}: Error finding container 80cf63c8938396fe82b71a82909ef11976a57c0461e7534ea813cdedf267e4e1: Status 404 returned error can't find the container with id 80cf63c8938396fe82b71a82909ef11976a57c0461e7534ea813cdedf267e4e1
Sep 17 21:52:45.144524 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:45.144479    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" event={"ID":"604fa74d-54bd-42ac-aae4-04f3e09e316c","Type":"ContainerStarted","Data":"294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2"}
Sep 17 21:52:45.144935 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:45.144575    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" event={"ID":"604fa74d-54bd-42ac-aae4-04f3e09e316c","Type":"ContainerStarted","Data":"80cf63c8938396fe82b71a82909ef11976a57c0461e7534ea813cdedf267e4e1"}
Sep 17 21:52:45.167298 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:45.167242    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" podStartSLOduration=2.167227938 podStartE2EDuration="2.167227938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:52:45.165229547 +0000 UTC m=+3815.744465410" watchObservedRunningTime="2026-09-17 21:52:45.167227938 +0000 UTC m=+3815.746463826"
Sep 17 21:52:46.019863 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:46.019827    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:54.019919 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:54.019881    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:54.031815 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:54.031793    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:52:54.195961 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:52:54.195932    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:53:17.738920 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:17.738883    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:53:17.750267 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:53:17.750228    2582 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 21:53:17.750267 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:53:17.750267 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:53:17.750267 ip-10-0-131-59 kubenswrapper[2582]:  > containerID="294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2" cmd=["/bin/sleep","15"]
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:17.750283    2582 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" message="" err=<
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:  >
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:53:17.750295    2582 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:         , stdout: , stderr: , exit code -1
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main" containerID="cri-o://294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2"
Sep 17 21:53:17.750616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:17.750340    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main" containerID="cri-o://294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2" gracePeriod=60
Sep 17 21:53:18.185560 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.185514    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 21:53:18.289556 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.289520    2582 generic.go:356] "Generic (PLEG): container finished" podID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerID="294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2" exitCode=0
Sep 17 21:53:18.289722 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.289597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" event={"ID":"604fa74d-54bd-42ac-aae4-04f3e09e316c","Type":"ContainerDied","Data":"294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2"}
Sep 17 21:53:18.488697 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.488670    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:53:18.583021 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.582984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583211 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583055    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bjfw\" (UniqueName: \"kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583211 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583211 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583211 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583171    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583211 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583192    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm\") pod \"604fa74d-54bd-42ac-aae4-04f3e09e316c\" (UID: \"604fa74d-54bd-42ac-aae4-04f3e09e316c\") "
Sep 17 21:53:18.583448 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home" (OuterVolumeSpecName: "home") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:53:18.583448 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:53:18.583562 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583449    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:18.583665 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.583643    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache" (OuterVolumeSpecName: "model-cache") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:53:18.585532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.585505    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm" (OuterVolumeSpecName: "dshm") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:53:18.585645 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.585572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:53:18.585645 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.585585    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw" (OuterVolumeSpecName: "kube-api-access-6bjfw") pod "604fa74d-54bd-42ac-aae4-04f3e09e316c" (UID: "604fa74d-54bd-42ac-aae4-04f3e09e316c"). InnerVolumeSpecName "kube-api-access-6bjfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:53:18.683954 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.683918    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjfw\" (UniqueName: \"kubernetes.io/projected/604fa74d-54bd-42ac-aae4-04f3e09e316c-kube-api-access-6bjfw\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:18.683954 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.683951    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:18.683954 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.683961    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/604fa74d-54bd-42ac-aae4-04f3e09e316c-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:18.684220 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.683969    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:18.684220 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:18.683977    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/604fa74d-54bd-42ac-aae4-04f3e09e316c-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:53:19.295340 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.295308    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"
Sep 17 21:53:19.295810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.295312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd" event={"ID":"604fa74d-54bd-42ac-aae4-04f3e09e316c","Type":"ContainerDied","Data":"80cf63c8938396fe82b71a82909ef11976a57c0461e7534ea813cdedf267e4e1"}
Sep 17 21:53:19.295810 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.295439    2582 scope.go:117] "RemoveContainer" containerID="294e1337f9dba297b41438574aa9942a9fe09d5b491ec479050d0c16fb4880b2"
Sep 17 21:53:19.330690 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.330662    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:53:19.332604 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.332578    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-8z4rd"]
Sep 17 21:53:19.583983 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:19.583904    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" path="/var/lib/kubelet/pods/604fa74d-54bd-42ac-aae4-04f3e09e316c/volumes"
Sep 17 21:53:35.134768 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.134679    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:53:35.135237 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.135182    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main"
Sep 17 21:53:35.135237 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.135201    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main"
Sep 17 21:53:35.135368 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.135302    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="604fa74d-54bd-42ac-aae4-04f3e09e316c" containerName="main"
Sep 17 21:53:35.139234 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.139210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.141920 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.141898    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:53:35.142284 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.142259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 21:53:35.142284 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.142274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 21:53:35.142535 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.142521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 21:53:35.142591 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.142578    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 21:53:35.142952 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.142936    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-w6qcl\""
Sep 17 21:53:35.156596 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.156574    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:53:35.312444 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrlc\" (UniqueName: \"kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.312628 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.312607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413190 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413190 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrlc\" (UniqueName: \"kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413408 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413702 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413454    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413814 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.413923 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.414019 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.413995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.414264 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.414244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.415748 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.415728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.415836 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.415747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.423931 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.423910    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrlc\" (UniqueName: \"kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc\") pod \"prestop-hook-test-kserve-764979c8fb-l2gwg\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.448844 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.448823    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:35.730420 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:35.730391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:53:35.732692 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:53:35.732656    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224bd928_3ab9_40a6_9e38_90df1efba112.slice/crio-a5588f2815e525d30139c94339eb1f620f93fba32e9c340f6b5821ca7af6d47b WatchSource:0}: Error finding container a5588f2815e525d30139c94339eb1f620f93fba32e9c340f6b5821ca7af6d47b: Status 404 returned error can't find the container with id a5588f2815e525d30139c94339eb1f620f93fba32e9c340f6b5821ca7af6d47b
Sep 17 21:53:36.372349 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:36.372305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerStarted","Data":"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"}
Sep 17 21:53:36.372803 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:36.372364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerStarted","Data":"a5588f2815e525d30139c94339eb1f620f93fba32e9c340f6b5821ca7af6d47b"}
Sep 17 21:53:38.388236 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:38.388198    2582 generic.go:356] "Generic (PLEG): container finished" podID="224bd928-3ab9-40a6-9e38-90df1efba112" containerID="a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7" exitCode=0
Sep 17 21:53:38.388609 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:38.388272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerDied","Data":"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"}
Sep 17 21:53:39.394730 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:39.394697    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerStarted","Data":"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"}
Sep 17 21:53:39.420576 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:39.420527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podStartSLOduration=2.295003733 podStartE2EDuration="4.420512136s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.125508403s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:53:39.417968826 +0000 UTC m=+3869.997204689" watchObservedRunningTime="2026-09-17 21:53:39.420512136 +0000 UTC m=+3869.999747999"
Sep 17 21:53:39.449079 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:39.449045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:45.449975 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:45.449935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:53:45.451361 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:45.451329    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:53:55.449591 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:53:55.449545    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:05.449368 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:05.449321    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:11.820433 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:11.820406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:54:11.826252 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:11.826231    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:54:11.830877 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:11.830852    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:54:11.835554 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:11.835535    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n5kq6_573bf01c-5c43-4076-a4bd-daaef1b49f84/ovn-acl-logging/0.log"
Sep 17 21:54:15.449619 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:15.449577    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:25.449687 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:25.449646    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:35.449938 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:35.449878    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:45.450007 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:45.449960    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:54:55.449741 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:54:55.449629    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:55:05.450053 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:05.449999    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:55:15.450180 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:15.450134    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:55:25.449512 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:25.449467    2582 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 21:55:35.459044 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:35.459004    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:55:35.466939 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:35.466910    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:55:39.722881 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:39.722843    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:55:54.745704 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:54.745656    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" containerID="cri-o://d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2" gracePeriod=45
Sep 17 21:55:55.495857 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.495828    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:55:55.659026 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.658931    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659026 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.658979    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659026 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659035    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddrlc\" (UniqueName: \"kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659160    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659211    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs\") pod \"224bd928-3ab9-40a6-9e38-90df1efba112\" (UID: \"224bd928-3ab9-40a6-9e38-90df1efba112\") "
Sep 17 21:55:55.659336 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659303    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache" (OuterVolumeSpecName: "model-cache") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:55.659652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659397    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:55:55.659652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659559    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-model-cache\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.659652 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659577    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/224bd928-3ab9-40a6-9e38-90df1efba112-cabundle-cert\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.659831 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.659803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home" (OuterVolumeSpecName: "home") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:55.661435 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.661403    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm" (OuterVolumeSpecName: "dshm") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:55.661584 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.661538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc" (OuterVolumeSpecName: "kube-api-access-ddrlc") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "kube-api-access-ddrlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:55:55.661837 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.661819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:55:55.671080 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.671047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:55.682532 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.682501    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "224bd928-3ab9-40a6-9e38-90df1efba112" (UID: "224bd928-3ab9-40a6-9e38-90df1efba112"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:55:55.760926 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760888    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrlc\" (UniqueName: \"kubernetes.io/projected/224bd928-3ab9-40a6-9e38-90df1efba112-kube-api-access-ddrlc\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.760926 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760922    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-kserve-provision-location\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.760926 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760933    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-tmp-dir\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.761371 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760943    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224bd928-3ab9-40a6-9e38-90df1efba112-tls-certs\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.761371 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760953    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-dshm\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:55.761371 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:55.760962    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224bd928-3ab9-40a6-9e38-90df1efba112-home\") on node \"ip-10-0-131-59.ec2.internal\" DevicePath \"\""
Sep 17 21:55:56.059608 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.059569    2582 generic.go:356] "Generic (PLEG): container finished" podID="224bd928-3ab9-40a6-9e38-90df1efba112" containerID="d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2" exitCode=0
Sep 17 21:55:56.059781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.059623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerDied","Data":"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"}
Sep 17 21:55:56.059781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.059668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" event={"ID":"224bd928-3ab9-40a6-9e38-90df1efba112","Type":"ContainerDied","Data":"a5588f2815e525d30139c94339eb1f620f93fba32e9c340f6b5821ca7af6d47b"}
Sep 17 21:55:56.059781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.059691    2582 scope.go:117] "RemoveContainer" containerID="d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"
Sep 17 21:55:56.059781 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.059629    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"
Sep 17 21:55:56.068990 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.068966    2582 scope.go:117] "RemoveContainer" containerID="a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"
Sep 17 21:55:56.076544 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.076525    2582 scope.go:117] "RemoveContainer" containerID="d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"
Sep 17 21:55:56.076816 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:55:56.076794    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2\": container with ID starting with d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2 not found: ID does not exist" containerID="d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"
Sep 17 21:55:56.076889 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.076828    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2"} err="failed to get container status \"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2\": rpc error: code = NotFound desc = could not find container \"d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2\": container with ID starting with d753cccff2490437d9890a870aa3e9e568e281c71e24b4e62a45a9ea76eef9b2 not found: ID does not exist"
Sep 17 21:55:56.076889 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.076853    2582 scope.go:117] "RemoveContainer" containerID="a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"
Sep 17 21:55:56.077147 ip-10-0-131-59 kubenswrapper[2582]: E0917 21:55:56.077126    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7\": container with ID starting with a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7 not found: ID does not exist" containerID="a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"
Sep 17 21:55:56.077216 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.077151    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7"} err="failed to get container status \"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7\": rpc error: code = NotFound desc = could not find container \"a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7\": container with ID starting with a3e257b8aa120a6c3275035b5c072ace35c6b4d44de7f99781488122f518b9c7 not found: ID does not exist"
Sep 17 21:55:56.082703 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.082677    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:55:56.086034 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.086012    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg"]
Sep 17 21:55:56.460669 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:56.460568    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-764979c8fb-l2gwg" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Sep 17 21:55:57.583568 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:55:57.583528    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" path="/var/lib/kubelet/pods/224bd928-3ab9-40a6-9e38-90df1efba112/volumes"
Sep 17 21:57:30.154165 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:30.154131    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7hpgj_0aae8fe1-d98a-471c-84c0-892b0b9b69ee/kuadrant-console-plugin/0.log"
Sep 17 21:57:30.263601 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:30.263566    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-2cfxd_9fdf113d-4de9-4ebf-8761-e0fed7caef90/manager/0.log"
Sep 17 21:57:30.273251 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:30.273225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9bthq_90a417e0-8995-4958-8030-5d8c44c9fd99/limitador/0.log"
Sep 17 21:57:34.358034 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.357997    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8tzf/must-gather-77rk7"]
Sep 17 21:57:34.358459 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.358396    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main"
Sep 17 21:57:34.358459 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.358409    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main"
Sep 17 21:57:34.358459 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.358417    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="storage-initializer"
Sep 17 21:57:34.358459 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.358424    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="storage-initializer"
Sep 17 21:57:34.358616 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.358489    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="224bd928-3ab9-40a6-9e38-90df1efba112" containerName="main"
Sep 17 21:57:34.361720 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.361704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.364659 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.364636    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"openshift-service-ca.crt\""
Sep 17 21:57:34.365708 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.365692    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8tzf\"/\"kube-root-ca.crt\""
Sep 17 21:57:34.366856 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.366838    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8tzf\"/\"default-dockercfg-4wxnw\""
Sep 17 21:57:34.373644 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.373619    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/must-gather-77rk7"]
Sep 17 21:57:34.498148 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.498090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65d1d4f9-6570-4073-8634-9ab4a455b77e-must-gather-output\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.498350 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.498159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tscc5\" (UniqueName: \"kubernetes.io/projected/65d1d4f9-6570-4073-8634-9ab4a455b77e-kube-api-access-tscc5\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.599395 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.599360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tscc5\" (UniqueName: \"kubernetes.io/projected/65d1d4f9-6570-4073-8634-9ab4a455b77e-kube-api-access-tscc5\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.599591 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.599443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65d1d4f9-6570-4073-8634-9ab4a455b77e-must-gather-output\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.599820 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.599789    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65d1d4f9-6570-4073-8634-9ab4a455b77e-must-gather-output\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.610725 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.610649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tscc5\" (UniqueName: \"kubernetes.io/projected/65d1d4f9-6570-4073-8634-9ab4a455b77e-kube-api-access-tscc5\") pod \"must-gather-77rk7\" (UID: \"65d1d4f9-6570-4073-8634-9ab4a455b77e\") " pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.672680 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.672649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/must-gather-77rk7"
Sep 17 21:57:34.944985 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.944958    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/must-gather-77rk7"]
Sep 17 21:57:34.946930 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:57:34.946884    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65d1d4f9_6570_4073_8634_9ab4a455b77e.slice/crio-5cb93833e4a2c1432c7e16cc37bd8a98c2c88826510b50168bffcddfb5080791 WatchSource:0}: Error finding container 5cb93833e4a2c1432c7e16cc37bd8a98c2c88826510b50168bffcddfb5080791: Status 404 returned error can't find the container with id 5cb93833e4a2c1432c7e16cc37bd8a98c2c88826510b50168bffcddfb5080791
Sep 17 21:57:34.948948 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:34.948930    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:57:35.506640 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:35.506605    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/must-gather-77rk7" event={"ID":"65d1d4f9-6570-4073-8634-9ab4a455b77e","Type":"ContainerStarted","Data":"5cb93833e4a2c1432c7e16cc37bd8a98c2c88826510b50168bffcddfb5080791"}
Sep 17 21:57:36.514015 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:36.513978    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/must-gather-77rk7" event={"ID":"65d1d4f9-6570-4073-8634-9ab4a455b77e","Type":"ContainerStarted","Data":"9c9d612133018e781d7f3c2a03c7cce1ee3b8969e11af6c4e5a779bff578a006"}
Sep 17 21:57:36.515841 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:36.515811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/must-gather-77rk7" event={"ID":"65d1d4f9-6570-4073-8634-9ab4a455b77e","Type":"ContainerStarted","Data":"f14c91800621916c41cec87d4890ea1f9d7ba9be2deead6f0fbea4e2d8c68cb6"}
Sep 17 21:57:36.532936 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:36.532876    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8tzf/must-gather-77rk7" podStartSLOduration=1.8308989150000001 podStartE2EDuration="2.532854067s" totalImagesPullingTime="701.955152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:36.531282118 +0000 UTC m=+4107.110517967" watchObservedRunningTime="2026-09-17 21:57:36.532854067 +0000 UTC m=+4107.112089919"
Sep 17 21:57:37.418697 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:37.418664    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fzp7r_3941b7de-2022-492f-a2fa-869389c08f96/global-pull-secret-syncer/0.log"
Sep 17 21:57:37.600302 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:37.600268    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t879j_a2f19303-dfd7-4544-8a8f-ca46440f2278/konnectivity-agent/0.log"
Sep 17 21:57:37.624745 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:37.624715    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-59.ec2.internal_3191f2c20200c49353fb9f7218658ab0/haproxy/0.log"
Sep 17 21:57:42.565951 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:42.565916    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7hpgj_0aae8fe1-d98a-471c-84c0-892b0b9b69ee/kuadrant-console-plugin/0.log"
Sep 17 21:57:42.790123 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:42.789120    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-2cfxd_9fdf113d-4de9-4ebf-8761-e0fed7caef90/manager/0.log"
Sep 17 21:57:42.811828 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:42.811799    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9bthq_90a417e0-8995-4958-8030-5d8c44c9fd99/limitador/0.log"
Sep 17 21:57:44.578440 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.578406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfgl7_14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e/node-exporter/0.log"
Sep 17 21:57:44.601012 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.600981    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfgl7_14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e/kube-rbac-proxy/0.log"
Sep 17 21:57:44.644384 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.644356    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zfgl7_14c2cacc-4fc5-4ef8-ba48-ad9f121f3f9e/init-textfile/0.log"
Sep 17 21:57:44.673857 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.673826    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mbxjw_bbd8b3fe-a405-4588-bdca-7237c11763b1/kube-rbac-proxy-main/0.log"
Sep 17 21:57:44.695217 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.695179    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mbxjw_bbd8b3fe-a405-4588-bdca-7237c11763b1/kube-rbac-proxy-self/0.log"
Sep 17 21:57:44.724538 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.724500    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-mbxjw_bbd8b3fe-a405-4588-bdca-7237c11763b1/openshift-state-metrics/0.log"
Sep 17 21:57:44.783978 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.783945    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/prometheus/0.log"
Sep 17 21:57:44.804010 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.803497    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/config-reloader/0.log"
Sep 17 21:57:44.835189 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.835162    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/thanos-sidecar/0.log"
Sep 17 21:57:44.868645 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.868611    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/kube-rbac-proxy-web/0.log"
Sep 17 21:57:44.895799 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.895679    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/kube-rbac-proxy/0.log"
Sep 17 21:57:44.933420 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.933391    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/kube-rbac-proxy-thanos/0.log"
Sep 17 21:57:44.959486 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:44.959456    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1966f40c-d5a1-450c-89b6-139c0dc8f098/init-config-reloader/0.log"
Sep 17 21:57:45.040690 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.040655    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-bqgv4_869782de-c32c-4d9b-aae6-2157b745bac4/prometheus-operator-admission-webhook/0.log"
Sep 17 21:57:45.176684 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.176600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/thanos-query/0.log"
Sep 17 21:57:45.201170 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.201126    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/kube-rbac-proxy-web/0.log"
Sep 17 21:57:45.223432 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.223388    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/kube-rbac-proxy/0.log"
Sep 17 21:57:45.262785 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.262752    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/prom-label-proxy/0.log"
Sep 17 21:57:45.283644 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.283565    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/kube-rbac-proxy-rules/0.log"
Sep 17 21:57:45.305405 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:45.305372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78ccfc565f-sbzqf_e5e784c6-209f-47c5-8bd0-c70fe73d6ab9/kube-rbac-proxy-metrics/0.log"
Sep 17 21:57:47.042739 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.042702    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/2.log"
Sep 17 21:57:47.049052 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.049024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-bdpxp_4f3e9553-733b-4bcf-bb90-e20d37ba7efb/console-operator/3.log"
Sep 17 21:57:47.257267 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.257223    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"]
Sep 17 21:57:47.262850 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.262819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.282342 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.282303    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"]
Sep 17 21:57:47.423428 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.423334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbcv\" (UniqueName: \"kubernetes.io/projected/0fdf827c-d089-4dc6-8197-b2f624a20610-kube-api-access-7kbcv\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.423428 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.423389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-proc\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.423674 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.423480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-sys\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.423674 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.423508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-podres\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.423674 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.423541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-lib-modules\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524107 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbcv\" (UniqueName: \"kubernetes.io/projected/0fdf827c-d089-4dc6-8197-b2f624a20610-kube-api-access-7kbcv\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524433 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-proc\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524623 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-proc\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524723 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-sys\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524723 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-podres\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524723 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-lib-modules\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.524927 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.524906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-lib-modules\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.525154 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.525127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-podres\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.525335 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.525318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fdf827c-d089-4dc6-8197-b2f624a20610-sys\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.534391 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.534358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbcv\" (UniqueName: \"kubernetes.io/projected/0fdf827c-d089-4dc6-8197-b2f624a20610-kube-api-access-7kbcv\") pod \"perf-node-gather-daemonset-bzrg9\" (UID: \"0fdf827c-d089-4dc6-8197-b2f624a20610\") " pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.573484 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.573444    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:47.608949 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.608898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-5thql_84f3a2ea-e3ea-449c-a292-7388c76c730a/download-server/0.log"
Sep 17 21:57:47.864520 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:47.864482    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"]
Sep 17 21:57:47.867531 ip-10-0-131-59 kubenswrapper[2582]: W0917 21:57:47.867497    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0fdf827c_d089_4dc6_8197_b2f624a20610.slice/crio-693f0c937527b92a1543a220f6da86c4e7464cc590992bcb1d3076feb5b03b87 WatchSource:0}: Error finding container 693f0c937527b92a1543a220f6da86c4e7464cc590992bcb1d3076feb5b03b87: Status 404 returned error can't find the container with id 693f0c937527b92a1543a220f6da86c4e7464cc590992bcb1d3076feb5b03b87
Sep 17 21:57:48.192599 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:48.192516    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-jpflf_d49208e2-0b60-4986-b5e9-674e612579e6/volume-data-source-validator/0.log"
Sep 17 21:57:48.593690 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:48.593655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9" event={"ID":"0fdf827c-d089-4dc6-8197-b2f624a20610","Type":"ContainerStarted","Data":"5d60d0fef9aebdefdec736344b05fb59a0a47097d061cba4614c7ad1f0952d90"}
Sep 17 21:57:48.593902 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:48.593710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9" event={"ID":"0fdf827c-d089-4dc6-8197-b2f624a20610","Type":"ContainerStarted","Data":"693f0c937527b92a1543a220f6da86c4e7464cc590992bcb1d3076feb5b03b87"}
Sep 17 21:57:48.593902 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:48.593738    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:48.613323 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:48.613269    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9" podStartSLOduration=1.613252972 podStartE2EDuration="1.613252972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:57:48.611396692 +0000 UTC m=+4119.190632544" watchObservedRunningTime="2026-09-17 21:57:48.613252972 +0000 UTC m=+4119.192488854"
Sep 17 21:57:49.198671 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:49.198639    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hlpm_4874db31-3321-40bf-8eed-2b111959c3ac/dns/0.log"
Sep 17 21:57:49.222866 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:49.222832    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8hlpm_4874db31-3321-40bf-8eed-2b111959c3ac/kube-rbac-proxy/0.log"
Sep 17 21:57:49.352353 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:49.352323    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-64s7w_3549c13d-d5d4-4694-bfe1-f4b0d1d069af/dns-node-resolver/0.log"
Sep 17 21:57:49.967880 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:49.967844    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2p4n6_4990bc2f-e0eb-4538-8273-f19cf0ac5339/node-ca/0.log"
Sep 17 21:57:51.739873 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:51.739839    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bsjht_445ed9f2-bb49-4023-ad4c-ef2b81082509/serve-healthcheck-canary/0.log"
Sep 17 21:57:52.360699 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:52.360673    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cnjp_0e7e13d1-3c12-48bf-b0a2-4212c7d05df2/kube-rbac-proxy/0.log"
Sep 17 21:57:52.388383 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:52.388346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cnjp_0e7e13d1-3c12-48bf-b0a2-4212c7d05df2/exporter/0.log"
Sep 17 21:57:52.410890 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:52.410859    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cnjp_0e7e13d1-3c12-48bf-b0a2-4212c7d05df2/extractor/0.log"
Sep 17 21:57:54.610915 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:54.610830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8tzf/perf-node-gather-daemonset-bzrg9"
Sep 17 21:57:55.702804 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:57:55.702776    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7dbd69c77d-q54kg_03abbed5-ba24-41a1-99b8-849fa2d7306d/manager/0.log"
Sep 17 21:58:03.181180 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:03.181151    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-mxr2k_a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec/migrator/0.log"
Sep 17 21:58:03.200402 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:03.200372    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-mxr2k_a4caaf28-5fa8-40c0-b139-15cb2c1ad9ec/graceful-termination/0.log"
Sep 17 21:58:04.619701 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.619670    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:58:04.638742 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.638716    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/egress-router-binary-copy/0.log"
Sep 17 21:58:04.659022 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.658996    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/cni-plugins/0.log"
Sep 17 21:58:04.678445 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.678418    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/bond-cni-plugin/0.log"
Sep 17 21:58:04.699585 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.699560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/routeoverride-cni/0.log"
Sep 17 21:58:04.719380 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.719351    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/whereabouts-cni-bincopy/0.log"
Sep 17 21:58:04.739370 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:04.739339    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2852r_465b7c1a-4120-4138-86bc-fbc4aa842ece/whereabouts-cni/0.log"
Sep 17 21:58:05.113958 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:05.113928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g4jlw_3797aa25-5a62-4ce0-8629-70912fb58b4e/kube-multus/0.log"
Sep 17 21:58:05.228571 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:05.228530    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-77s9j_5e9414f0-9f8a-4d7c-bc9d-527f06ff7927/network-metrics-daemon/0.log"
Sep 17 21:58:05.246818 ip-10-0-131-59 kubenswrapper[2582]: I0917 21:58:05.246790    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-77s9j_5e9414f0-9f8a-4d7c-bc9d-527f06ff7927/kube-rbac-proxy/0.log"