Sep 11 16:16:56.233711 ip-10-0-128-144 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 16:16:56.635317 ip-10-0-128-144 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:56.635317 ip-10-0-128-144 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 16:16:56.635317 ip-10-0-128-144 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:56.635317 ip-10-0-128-144 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 16:16:56.635317 ip-10-0-128-144 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 16:16:56.636448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.636347    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 16:16:56.642334 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642315    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.642334 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642336    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642340    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642343    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642347    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642350    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642354    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642358    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642364    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642368    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642370    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642374    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642376    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642379    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642382    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642386    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642389    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642392    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642407    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.642418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642412    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642416    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642419    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642423    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642425    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642431    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642434    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642437    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642439    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642443    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642448    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642451    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642454    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642456    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642459    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642461    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642464    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642466    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642469    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.642886 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642471    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642474    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642476    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642479    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642481    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642483    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642487    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642489    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642492    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642494    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642497    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642499    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642504    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642507    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642511    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642514    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642519    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642522    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.643369 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642525    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642527    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642530    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642533    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642536    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642539    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642541    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642544    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642546    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642549    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642551    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642554    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642556    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642559    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642561    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642564    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642568    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642570    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642573    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.643846 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642575    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642579    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642581    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642584    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642586    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642589    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642591    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642594    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642598    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642601    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642603    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642606    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642609    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642611    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642614    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642616    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642619    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642622    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642625    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642627    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.644305 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.642634    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644232    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644241    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644245    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644249    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644253    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644256    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644258    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644261    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644264    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644266    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644269    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644271    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644274    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644277    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644280    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644282    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644285    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644287    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.644804 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644290    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644293    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644296    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644299    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644301    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644304    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644307    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644309    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644312    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644315    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644317    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644320    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644323    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644326    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644328    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644331    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644333    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644337    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644340    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.645321 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644343    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644346    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644348    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644351    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644354    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644357    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644360    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644363    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644366    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644369    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644371    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644374    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644377    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644380    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644383    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644386    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644388    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644391    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644394    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644408    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.645811 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644411    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644413    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644416    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644418    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644421    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644424    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644426    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644429    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644432    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644434    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644437    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644439    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644442    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644444    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644447    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644450    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644454    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644457    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644459    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.646303 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644462    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644464    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644468    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644470    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644473    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644476    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644481    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644485    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644487    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644490    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644493    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644496    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644498    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644500    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644503    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644506    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644508    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644511    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644514    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644517    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.646814 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644519    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644522    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644929    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644934    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644937    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644939    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644942    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644945    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644948    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644951    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644953    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644958    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644962    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644965    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644969    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644972    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644976    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644979    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644982    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.647308 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644985    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644988    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644990    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644993    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644996    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.644998    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645001    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645003    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645006    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645010    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645013    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645016    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645019    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645022    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645027    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645030    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645033    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645037    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645039    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.647803 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645042    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645045    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645047    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645050    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645052    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645055    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645058    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645060    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645063    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645066    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645069    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645072    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645074    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645077    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645079    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645081    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645084    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645086    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645089    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645091    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.648284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645094    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645096    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645099    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645101    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645104    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645106    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645109    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645111    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645114    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645116    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645120    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645123    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645125    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645127    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645130    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645132    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645135    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645138    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645140    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645143    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.648792 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645145    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645148    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645151    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645153    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645155    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645158    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645160    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645163    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645165    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645167    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645170    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645172    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645175    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645177    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645180    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645182    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645184    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645187    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645189    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645192    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.649279 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645194    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645197    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.645199    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645282    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645323    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645332    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645337    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645341    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645344    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645349    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645354    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645357    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645360    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645364    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645367    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645378    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645381    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645384    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645387    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645390    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645393    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645412    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645416    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645421    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 16:16:56.649857 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645424    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645428    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645432    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645435    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645438    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645441    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645444    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645448    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645451    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645454    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645457    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645461    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645465    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645468    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645471    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645474    2563 flags.go:64] FLAG: --enable-server="true"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645477    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645485    2563 flags.go:64] FLAG: --event-burst="100"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645488    2563 flags.go:64] FLAG: --event-qps="50"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645491    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645494    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645497    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645501    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645503    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645508    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 16:16:56.650478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645511    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645514    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645517    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645520    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645523    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645525    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645529    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645531    2563 flags.go:64] FLAG: --feature-gates=""
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645535    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645538    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645541    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645545    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645548    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645551    2563 flags.go:64] FLAG: --help="false"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645556    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645559    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645562    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645565    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645582    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645586    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645589    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645592    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645595    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 16:16:56.651096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645598    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645601    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645604    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645607    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645610    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645613    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645616    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645619    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645621    2563 flags.go:64] FLAG: --lock-file=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645625    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645628    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645631    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645636    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645639    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645642    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645645    2563 flags.go:64] FLAG: --logging-format="text"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645647    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645651    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645653    2563 flags.go:64] FLAG: --manifest-url=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645656    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645661    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645663    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645668    2563 flags.go:64] FLAG: --max-pods="110"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645673    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645676    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 16:16:56.651698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645679    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645682    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645685    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645688    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645691    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645699    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645704    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645707    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645710    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645713    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645718    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645721    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645724    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645727    2563 flags.go:64] FLAG: --port="10250"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645730    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645733    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0161bd45b19ae63b3"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645736    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645740    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645743    2563 flags.go:64] FLAG: --register-node="true"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645746    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645749    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645752    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645755    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645758    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645761    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 16:16:56.652313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645764    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645767    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645770    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645773    2563 flags.go:64] FLAG: --runonce="false"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645776    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645780    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645783    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645786    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645789    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645792    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645796    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645799    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645802    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645805    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645809    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645812    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645816    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645819    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645822    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645827    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645830    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645833    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645840    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645842    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645845    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 16:16:56.652935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645849    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645852    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645855    2563 flags.go:64] FLAG: --v="2"
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645860    2563 flags.go:64] FLAG: --version="false"
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645864    2563 flags.go:64] FLAG: --vmodule=""
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645867    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.645870    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646009    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646013    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646016    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646020    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646024    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646030    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646033    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646035    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646038    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646042    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646046    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646049    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646051    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646054    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.653611 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646058    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646060    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646063    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646066    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646068    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646071    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646074    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646076    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646079    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646081    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646084    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646087    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646090    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646093    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646096    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646098    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646101    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646103    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646106    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646108    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.654185 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646111    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646114    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646116    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646120    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646123    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646126    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646128    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646131    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646133    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646136    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646149    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646153    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646157    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646160    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646163    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646167    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646170    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646173    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646176    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.654689 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646179    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646182    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646184    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646187    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646189    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646192    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646196    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646198    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646201    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646204    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646207    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646209    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646212    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646214    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646217    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646220    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646224    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646227    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646229    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646232    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.655201 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646235    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646237    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646240    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646242    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646245    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646249    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646251    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646254    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646256    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646259    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646262    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646264    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646267    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646270    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646273    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646275    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646278    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646280    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646283    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646286    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.655719 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646289    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646291    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646294    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646296    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646299    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.646301    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.656223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.646943    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:56.658470 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.658447    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 16:16:56.658470 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.658467    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 16:16:56.659439 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.659426    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 16:16:56.659493 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.659447    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 16:16:56.659546 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659540    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659549    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659556    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659562    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659568    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659573    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659577    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659582    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659587    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659592    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659596    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.659597 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659602    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659607    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659611    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659615    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659620    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659624    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659628    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659632    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659636    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659641    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659647    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659651    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659654    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659658    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659663    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659667    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659671    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659675    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659679    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659684    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.660082 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659688    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659692    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659696    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659703    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659709    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659714    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659719    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659723    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659728    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659732    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659739    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659746    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659750    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659754    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659759    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659765    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659771    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659777    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659783    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.660984 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659788    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659794    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659798    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659804    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659808    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659812    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659816    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659821    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659825    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659829    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659833    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659837    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659841    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659846    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659850    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659854    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659858    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659863    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659867    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659871    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.661497 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659876    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659880    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659885    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659889    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659893    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659898    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659902    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659907    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659911    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659918    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659923    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659927    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659932    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659936    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659940    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659945    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659949    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659953    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659957    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.662231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659961    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659965    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659970    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659974    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659979    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659983    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659987    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659991    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.659995    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660000    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.660007    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660202    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660211    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660216    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660220    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 16:16:56.662809 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660224    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660229    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660236    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660241    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660245    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660251    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660255    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660260    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660265    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660270    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660274    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660279    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660283    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660287    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660291    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660296    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660300    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660304    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660309    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 16:16:56.663293 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660313    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660317    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660321    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660326    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660330    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660334    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660341    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660346    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660351    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660355    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660361    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660365    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660370    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660375    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660380    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660384    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660388    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660393    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660414    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660419    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 16:16:56.664117 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660425    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660430    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660436    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660441    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660446    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660450    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660454    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660458    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660462    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660467    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660471    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660475    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660480    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660484    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660489    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660493    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660497    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660502    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660506    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660510    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 16:16:56.664671 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660514    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660518    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660522    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660527    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660531    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660536    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660540    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660544    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660548    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660552    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660557    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660561    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660566    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660570    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660575    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660580    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660584    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660588    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660592    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660596    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 16:16:56.665297 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660601    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660606    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660610    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660614    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660618    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660622    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660627    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660631    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660635    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660639    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660644    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660648    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660652    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660656    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660661    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 16:16:56.665805 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:56.660668    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 16:16:56.666209 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.660676    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 16:16:56.666209 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.660853    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 16:16:56.666209 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.664935    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 16:16:56.666209 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.666135    2563 server.go:1019] "Starting client certificate rotation"
Sep 11 16:16:56.666328 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.666248    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:56.666328 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.666279    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 16:16:56.669170 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.669152    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:56.673026 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.673008    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 16:16:56.681069 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.681053    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 11 16:16:56.684726 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.684712    2563 log.go:25] "Validated CRI v1 image API"
Sep 11 16:16:56.685526 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.685510    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 16:16:56.689889 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.689860    2563 fs.go:135] Filesystem UUIDs: map[553d9ddd-68af-4ce0-9743-885bc47e5d3e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c184c814-ef06-45a5-bf2f-e62b2e6b6567:/dev/nvme0n1p4]
Sep 11 16:16:56.689963 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.689888    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 16:16:56.695855 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.695731    2563 manager.go:216] Machine: {Timestamp:2026-09-11 16:16:56.695235269 +0000 UTC m=+0.363415609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102379 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a78b9d2c4cea66117d864d9572f3 SystemUUID:ec29a78b-9d2c-4cea-6611-7d864d9572f3 BootID:9b5e9ace-979d-4819-9156-d9b84afd937d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:61:0d:3e:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:61:0d:3e:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:c5:8e:8a:06:29 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 16:16:56.695855 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.695840    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 16:16:56.696024 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.695946    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 16:16:56.697924 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.697904    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:56.699671 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.699641    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 16:16:56.699843 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.699678    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 16:16:56.701432 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.701419    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 16:16:56.701498 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.701448    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 16:16:56.701498 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.701460    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:56.702446 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.702434    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 16:16:56.703661 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.703650    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 16:16:56.708949 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.708935    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:56.709098 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709087    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 16:16:56.709154 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709125    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 16:16:56.709154 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709144    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 16:16:56.709251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709164    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 16:16:56.709251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709179    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 11 16:16:56.709251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.709191    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 16:16:56.712292 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.712260    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:56.712531 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.712516    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 16:16:56.714999 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.714980    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 16:16:56.717692 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.717670    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 16:16:56.717749 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.717673    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 16:16:56.718090 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.718078    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 16:16:56.718137 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.718107    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 16:16:56.720092 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720080    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720097    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720103    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720108    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720118    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720123    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720136    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 16:16:56.720144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720141    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 16:16:56.720326 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720148    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 16:16:56.720326 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720153    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 16:16:56.720326 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720162    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 16:16:56.720326 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.720167    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 16:16:56.721041 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.721031    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 16:16:56.721041 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.721041    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 16:16:56.725074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725060    2563 server.go:1268] "Started kubelet"
Sep 11 16:16:56.725249 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725200    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 16:16:56.725321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725234    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 16:16:56.725321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725275    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 16:16:56.725794 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725768    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 16:16:56.725860 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.725846    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfrbz"
Sep 11 16:16:56.725989 ip-10-0-128-144 systemd[1]: Started Kubernetes Kubelet.
Sep 11 16:16:56.727184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.727168    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 16:16:56.728192 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.728179    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 16:16:56.731851 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.731831    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 16:16:56.732776 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.731707    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-144.ec2.internal.18d450754e1fd2cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-144.ec2.internal,UID:ip-10-0-128-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-144.ec2.internal,},FirstTimestamp:2026-09-11 16:16:56.725033676 +0000 UTC m=+0.393214014,LastTimestamp:2026-09-11 16:16:56.725033676 +0000 UTC m=+0.393214014,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-144.ec2.internal,}"
Sep 11 16:16:56.733874 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.733857    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfrbz"
Sep 11 16:16:56.736038 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.736025    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:56.736737 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.736716    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 16:16:56.737089 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.737070    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:56.737597 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.737579    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 16:16:56.737597 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.737581    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 16:16:56.737718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.737606    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 16:16:56.737718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.737681    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 16:16:56.737718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.737692    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 16:16:56.738513 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738498    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 16:16:56.738513 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738512    2563 factory.go:55] Registering systemd factory
Sep 11 16:16:56.738625 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738531    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 11 16:16:56.738830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738777    2563 factory.go:153] Registering CRI-O factory
Sep 11 16:16:56.738830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738804    2563 factory.go:223] Registration of the crio container factory successfully
Sep 11 16:16:56.738830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738824    2563 factory.go:103] Registering Raw factory
Sep 11 16:16:56.738978 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.738840    2563 manager.go:1196] Started watching for new ooms in manager
Sep 11 16:16:56.739229 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.739218    2563 manager.go:349] Starting recovery of all containers
Sep 11 16:16:56.745269 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.745064    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:56.746241 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.746211    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 16:16:56.747859 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.747834    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:56.754664 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.754543    2563 manager.go:354] Recovery completed
Sep 11 16:16:56.758708 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.758691    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.760499 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.760482    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.760585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.760513    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.760585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.760529    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.761198 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.761181    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 16:16:56.761198 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.761192    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.761198 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.761196    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 16:16:56.761355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.761215    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 16:16:56.761983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.761965    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.762044 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.762000    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.762044 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.762015    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.837695 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.837658    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:56.861050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.861017    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 16:16:56.861192 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.861082    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 16:16:56.861192 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.861108    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 16:16:56.861285 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.861205    2563 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 11 16:16:56.863210 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.863191    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:56.900101 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.900015    2563 policy_none.go:49] "None policy: Start"
Sep 11 16:16:56.900101 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.900042    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 16:16:56.900101 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.900055    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 16:16:56.902215 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.902202    2563 policy_none.go:47] "Start"
Sep 11 16:16:56.937886 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.937862    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:56.938000 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.937967    2563 manager.go:337] "Starting Device Plugin manager"
Sep 11 16:16:56.938054 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.938012    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 16:16:56.938054 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938023    2563 server.go:87] "Starting device plugin registration server"
Sep 11 16:16:56.938454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938443    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 16:16:56.938507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938455    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 16:16:56.938583 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938567    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 16:16:56.938697 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938655    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 16:16:56.938697 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.938668    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 16:16:56.939273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.939257    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 16:16:56.939342 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.939296    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:56.962232 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.962190    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"]
Sep 11 16:16:56.962351 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.962277    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.963836 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.963822    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.963902 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.963849    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.963902 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.963859    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.966195 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966183    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.966321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:56.966364 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966337    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.966938 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966921    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.966938 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966934    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.967076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966949    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.967076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966958    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.967076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966962    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.967076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.966969    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.971322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.971307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:56.971392 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.971332    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:56.972079 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.972064    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:56.972179 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.972089    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:56.972179 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:56.972101    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:56.988024 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.988000    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:56.993603 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:56.993586    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-144.ec2.internal\" not found" node="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.038878 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.038854    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 16:16:57.039040 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.038898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.039040 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.038928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.039040 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.038954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.039837 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.039823    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 16:16:57.039923 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.039850    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 16:16:57.039923 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.039865    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeHasSufficientPID"
Sep 11 16:16:57.039923 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.039897    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.047548 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.047534    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.047617 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:57.047556    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-144.ec2.internal\": node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:57.068234 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:57.068207    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:57.139239 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.139355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.139355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.139355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37310a8518ae4b4c8f438099a305077e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-144.ec2.internal\" (UID: \"37310a8518ae4b4c8f438099a305077e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.139355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.139355 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:57.139318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a022f343c02e056acab4e9c92407e8c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal\" (UID: \"0a022f343c02e056acab4e9c92407e8c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:57.168351 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:57.168274    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.443328 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.443285    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.465477 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.465449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:58.470458 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.470448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:58.543597 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.543544    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.644189 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.644104    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.744617 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.744572    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.840141 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.840113    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 16:16:58.840609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.840248    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:16:58.840609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.840303    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 16:16:58.845354 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.845333    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:58.911662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.911582    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 16:11:57 +0000 UTC" deadline="2028-02-20 03:39:19.584368121 +0000 UTC"
Sep 11 16:16:58.911662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:58.911610    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12635h22m20.672761087s"
Sep 11 16:16:58.946272 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:58.946241    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.046818 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.046783    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.096651 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:59.096607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37310a8518ae4b4c8f438099a305077e.slice/crio-b6000e92de9129e9d726f21fdae2bcb2d477e3ca54aedf706fc0021804440467 WatchSource:0}: Error finding container b6000e92de9129e9d726f21fdae2bcb2d477e3ca54aedf706fc0021804440467: Status 404 returned error can't find the container with id b6000e92de9129e9d726f21fdae2bcb2d477e3ca54aedf706fc0021804440467
Sep 11 16:16:59.096997 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:16:59.096976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a022f343c02e056acab4e9c92407e8c.slice/crio-1ab9a6a654522796e90f3d483569ed955f26bf3094ecdb9404021dd5a66b230a WatchSource:0}: Error finding container 1ab9a6a654522796e90f3d483569ed955f26bf3094ecdb9404021dd5a66b230a: Status 404 returned error can't find the container with id 1ab9a6a654522796e90f3d483569ed955f26bf3094ecdb9404021dd5a66b230a
Sep 11 16:16:59.103237 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.103211    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:16:59.147121 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.147085    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.247669 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.247592    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.318864 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.318830    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:59.348405 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.348369    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.449116 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.449083    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-144.ec2.internal\" not found"
Sep 11 16:16:59.477441 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.477412    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:59.511830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.511741    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:59.523539 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.523512    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:59.524314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.524300    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal"
Sep 11 16:16:59.537060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.537033    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 16:16:59.700921 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.700889    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:16:59.884893 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.884817    2563 apiserver.go:52] "Watching apiserver"
Sep 11 16:16:59.890611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.890583    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 16:16:59.891081 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.891055    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-rx84r","openshift-multus/network-metrics-daemon-zjf55","openshift-network-diagnostics/network-check-target-tb8lh","openshift-network-operator/iptables-alerter-rjnjb","kube-system/konnectivity-agent-f8rwn","openshift-cluster-node-tuning-operator/tuned-bz7hh","openshift-dns/node-resolver-g5nf4","openshift-image-registry/node-ca-mjnxd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal","openshift-multus/multus-hbsz8","openshift-ovn-kubernetes/ovnkube-node-shkzc","kube-system/global-pull-secret-syncer-prnmx","kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"]
Sep 11 16:16:59.893577 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.893550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:16:59.895929 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.895907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.896515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.896491    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.896515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.896506    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.896515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.896514    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8qh2x\""
Sep 11 16:16:59.898212 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 16:16:59.898886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 16:16:59.898886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.898886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 16:16:59.898886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hxbmp\""
Sep 11 16:16:59.898886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.898840    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.899344 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.899323    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:16:59.899467 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.899436    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:16:59.899616 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.899600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 16:16:59.901918 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.901898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:16:59.904421 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.904399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:16:59.904609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.904592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 16:16:59.904728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.904645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q6jbx\""
Sep 11 16:16:59.904728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.904593    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.904827 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.904807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.906889 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.906868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.907317 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.907244    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 16:16:59.907742 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.907560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 16:16:59.907976 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.907896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gcbzg\""
Sep 11 16:16:59.909769 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.909288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.909769 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.909583    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.910206 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.910172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kjtcq\""
Sep 11 16:16:59.911884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.911071    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 16:16:59.911884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.911849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.913537 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.912107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:16:59.913537 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.912586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.913537 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.912964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.913537 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.913113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sqgqx\""
Sep 11 16:16:59.913997 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.913820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 16:16:59.914983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.914746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:16:59.914983 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.914828    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:16:59.915847 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.915826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 16:16:59.915932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.915846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.915972 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.915942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.916211 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.916189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmllf\""
Sep 11 16:16:59.917478 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.917190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hbsz8"
Sep 11 16:16:59.919821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.919662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 16:16:59.919821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.919662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 16:16:59.919964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.919923    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 16:16:59.919964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.919942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 16:16:59.919964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.919945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9r49b\""
Sep 11 16:16:59.920796 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.920779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:16:59.921508 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-netns\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-var-lib-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-konnectivity-ca\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:16:59.921596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-modprobe-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.921596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-var-lib-kubelet\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-tuned\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-slash\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-ovn\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-run\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovn-node-metrics-cert\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-script-lib\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-systemd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.921821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-bin\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-kube-api-access-6st28\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a478cd40-13f0-40f6-8d19-12af35e82945-serviceca\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-conf\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-lib-modules\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.921978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-iptables-alerter-script\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-socket-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-env-overrides\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-tmp\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-kubernetes\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/536a788b-d062-40b8-9243-8c171bd82b95-kube-api-access-m77bp\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.922169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xqm\" (UniqueName: \"kubernetes.io/projected/a478cd40-13f0-40f6-8d19-12af35e82945-kube-api-access-v2xqm\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-tmp-dir\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-node-log\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-log-socket\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-config\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-sys\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-agent-certs\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-systemd\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-host\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zfh\" (UniqueName: \"kubernetes.io/projected/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-kube-api-access-n5zfh\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-kubelet\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-systemd-units\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.922700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a478cd40-13f0-40f6-8d19-12af35e82945-host\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tp8\" (UniqueName: \"kubernetes.io/projected/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-kube-api-access-89tp8\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-sys-fs\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-netd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9mt\" (UniqueName: \"kubernetes.io/projected/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-kube-api-access-zt9mt\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-etc-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-registration-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-device-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.922990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-hosts-file\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysconfig\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8574\" (UniqueName: \"kubernetes.io/projected/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-kube-api-access-q8574\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-host-slash\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:16:59.923420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5h2pr\""
Sep 11 16:16:59.924215 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:16:59.923361    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:16:59.924215 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.923466    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 16:16:59.947690 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.947645    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-68778"
Sep 11 16:16:59.957149 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:16:59.957120    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-68778"
Sep 11 16:17:00.013221 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.013190    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 16:17:00.023331 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-agent-certs\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-systemd\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-host\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkvf\" (UniqueName: \"kubernetes.io/projected/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-kube-api-access-fmkvf\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-multus-certs\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-kubelet-config\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-host\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.023464    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-etc-selinux\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.023585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zfh\" (UniqueName: \"kubernetes.io/projected/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-kube-api-access-n5zfh\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-kubelet\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.023619    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:00.523592446 +0000 UTC m=+3.017492238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-kubelet\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-systemd-units\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-systemd-units\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a478cd40-13f0-40f6-8d19-12af35e82945-host\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tp8\" (UniqueName: \"kubernetes.io/projected/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-kube-api-access-89tp8\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-system-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-sys-fs\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cnibin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-k8s-cni-cncf-io\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024159 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.023997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-netd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-bin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-sys-fs\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024056    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a478cd40-13f0-40f6-8d19-12af35e82945-host\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-etc-kubernetes\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-netd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcj9\" (UniqueName: \"kubernetes.io/projected/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-kube-api-access-brcj9\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9mt\" (UniqueName: \"kubernetes.io/projected/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-kube-api-access-zt9mt\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-etc-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-dbus\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-registration-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-device-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-hosts-file\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysconfig\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.024932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8574\" (UniqueName: \"kubernetes.io/projected/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-kube-api-access-q8574\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-daemon-config\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-host-slash\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-hostroot\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-system-cni-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-netns\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-var-lib-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysconfig\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-konnectivity-ca\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-modprobe-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-etc-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-var-lib-kubelet\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-tuned\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-slash\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-registration-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-ovn\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-ovn\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.025896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-run\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cni-binary-copy\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cnibin\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovn-node-metrics-cert\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-script-lib\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-systemd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-bin\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-kubelet\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-kube-api-access-6st28\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.024993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-device-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a478cd40-13f0-40f6-8d19-12af35e82945-serviceca\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-hosts-file\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-systemd\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-conf\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-lib-modules\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-conf-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.026725 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-iptables-alerter-script\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-socket-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-env-overrides\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-tmp\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-socket-dir-parent\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-kubernetes\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-netns\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-binary-copy\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/536a788b-d062-40b8-9243-8c171bd82b95-kube-api-access-m77bp\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a478cd40-13f0-40f6-8d19-12af35e82945-serviceca\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xqm\" (UniqueName: \"kubernetes.io/projected/a478cd40-13f0-40f6-8d19-12af35e82945-kube-api-access-v2xqm\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-tmp-dir\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-konnectivity-ca\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-node-log\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-host-slash\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.027519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-log-socket\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-config\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-var-lib-kubelet\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-var-lib-openvswitch\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-sys\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-slash\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-multus\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-run\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-sys\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-kubernetes\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-script-lib\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-run-systemd\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-cni-bin\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.028883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.026975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-conf\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.027054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-lib-modules\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.027556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-iptables-alerter-script\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.027662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/536a788b-d062-40b8-9243-8c171bd82b95-socket-dir\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-node-log\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-env-overrides\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-log-socket\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-modprobe-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-tmp-dir\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.025666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-host-run-netns\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-sysctl-d\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.029750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.028598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovnkube-config\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.030875 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.030578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-etc-tuned\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.030875 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.030661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-ovn-node-metrics-cert\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.030875 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.030858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-tmp\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.031158 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.030874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73d5d9f0-49b4-4ac7-ab61-a3dd59301b65-agent-certs\") pod \"konnectivity-agent-f8rwn\" (UID: \"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65\") " pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:00.032931 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.032908    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:00.033020 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.032936    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:00.033020 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.032952    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:00.033190 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.033024    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:00.533004913 +0000 UTC m=+3.026904705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:00.034729 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.034651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9mt\" (UniqueName: \"kubernetes.io/projected/72e00e16-c3b3-4c18-b68f-f491df7b9a8f-kube-api-access-zt9mt\") pod \"node-resolver-g5nf4\" (UID: \"72e00e16-c3b3-4c18-b68f-f491df7b9a8f\") " pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.035019 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.034993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tp8\" (UniqueName: \"kubernetes.io/projected/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-kube-api-access-89tp8\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:00.035178 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.035157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zfh\" (UniqueName: \"kubernetes.io/projected/04975c24-12e2-4cfc-a67e-f7b83f2ce37a-kube-api-access-n5zfh\") pod \"ovnkube-node-shkzc\" (UID: \"04975c24-12e2-4cfc-a67e-f7b83f2ce37a\") " pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.036254 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.036215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77bp\" (UniqueName: \"kubernetes.io/projected/536a788b-d062-40b8-9243-8c171bd82b95-kube-api-access-m77bp\") pod \"aws-ebs-csi-driver-node-kfkvn\" (UID: \"536a788b-d062-40b8-9243-8c171bd82b95\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.036933 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.036902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6st28\" (UniqueName: \"kubernetes.io/projected/a88ca20e-3919-44b3-b0d7-18ae7a2f1c97-kube-api-access-6st28\") pod \"iptables-alerter-rjnjb\" (UID: \"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97\") " pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.039729 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.039708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xqm\" (UniqueName: \"kubernetes.io/projected/a478cd40-13f0-40f6-8d19-12af35e82945-kube-api-access-v2xqm\") pod \"node-ca-mjnxd\" (UID: \"a478cd40-13f0-40f6-8d19-12af35e82945\") " pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.039943 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.039918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8574\" (UniqueName: \"kubernetes.io/projected/d6a89602-b607-47e9-bd5d-8c2a5daa01fb-kube-api-access-q8574\") pod \"tuned-bz7hh\" (UID: \"d6a89602-b607-47e9-bd5d-8c2a5daa01fb\") " pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.040908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.040858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerStarted","Data":"1ab9a6a654522796e90f3d483569ed955f26bf3094ecdb9404021dd5a66b230a"}
Sep 11 16:17:00.042245 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.042216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" event={"ID":"37310a8518ae4b4c8f438099a305077e","Type":"ContainerStarted","Data":"b6000e92de9129e9d726f21fdae2bcb2d477e3ca54aedf706fc0021804440467"}
Sep 11 16:17:00.126959 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.126924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-conf-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.126959 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.126964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-socket-dir-parent\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.126989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-conf-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-socket-dir-parent\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-netns\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-netns\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-binary-copy\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.127166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-multus\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkvf\" (UniqueName: \"kubernetes.io/projected/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-kube-api-access-fmkvf\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-multus\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-multus-certs\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.127279    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-kubelet-config\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-system-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.127345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:00.627327076 +0000 UTC m=+3.121226900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-multus-certs\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-system-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-kubelet-config\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cnibin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cnibin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-k8s-cni-cncf-io\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.127509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-bin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-run-k8s-cni-cncf-io\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-etc-kubernetes\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcj9\" (UniqueName: \"kubernetes.io/projected/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-kube-api-access-brcj9\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-cni-bin\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-dbus\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-etc-kubernetes\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-daemon-config\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-dbus\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-hostroot\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-hostroot\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-system-cni-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-system-cni-dir\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-binary-copy\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cni-binary-copy\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.128327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cnibin\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cnibin\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-kubelet\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-cni-dir\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.127933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-host-var-lib-kubelet\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.128070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-multus-daemon-config\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.129145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.128203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-cni-binary-copy\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.137146 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.137088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcj9\" (UniqueName: \"kubernetes.io/projected/b9c3fd3d-faf5-4651-ba28-8b10cb47243c-kube-api-access-brcj9\") pod \"multus-hbsz8\" (UID: \"b9c3fd3d-faf5-4651-ba28-8b10cb47243c\") " pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.137489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.137469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkvf\" (UniqueName: \"kubernetes.io/projected/0ede9424-cc8b-4a73-b30f-8bcf98e8e129-kube-api-access-fmkvf\") pod \"multus-additional-cni-plugins-rx84r\" (UID: \"0ede9424-cc8b-4a73-b30f-8bcf98e8e129\") " pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.178300 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.178275    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 16:17:00.205755 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.205724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5nf4"
Sep 11 16:17:00.215055 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.215032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:00.224874 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.224849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rjnjb"
Sep 11 16:17:00.238841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.238816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:00.245579 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.245562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bz7hh"
Sep 11 16:17:00.252173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.252155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn"
Sep 11 16:17:00.261749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.261729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjnxd"
Sep 11 16:17:00.269410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.269385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hbsz8"
Sep 11 16:17:00.275010 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.274988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rx84r"
Sep 11 16:17:00.531320 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.531231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:00.531478 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.531400    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:00.531478 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.531476    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:01.531457252 +0000 UTC m=+4.025357057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:00.632320 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.632284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:00.632485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.632331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:00.632485 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632435    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:00.632485 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632453    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:00.632485 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632470    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:00.632485 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632483    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:00.632751 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632501    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:01.632483583 +0000 UTC m=+4.126383388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:00.632751 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:00.632519    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:01.632508412 +0000 UTC m=+4.126408218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:00.960334 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.960241    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:59 +0000 UTC" deadline="2028-04-20 05:33:30.789870414 +0000 UTC"
Sep 11 16:17:00.960334 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:00.960290    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14077h16m29.829583636s"
Sep 11 16:17:00.999171 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:00.999020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a89602_b607_47e9_bd5d_8c2a5daa01fb.slice/crio-19f997ff13146f1ec626de2f96c8c4a5efae2c26d6830c4759e374f767c4e5d3 WatchSource:0}: Error finding container 19f997ff13146f1ec626de2f96c8c4a5efae2c26d6830c4759e374f767c4e5d3: Status 404 returned error can't find the container with id 19f997ff13146f1ec626de2f96c8c4a5efae2c26d6830c4759e374f767c4e5d3
Sep 11 16:17:01.000746 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:01.000642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e00e16_c3b3_4c18_b68f_f491df7b9a8f.slice/crio-c5f4b806b776b9602bfeb5bba45e064d8a5cd86acaab468a5bf3ff42af7a376f WatchSource:0}: Error finding container c5f4b806b776b9602bfeb5bba45e064d8a5cd86acaab468a5bf3ff42af7a376f: Status 404 returned error can't find the container with id c5f4b806b776b9602bfeb5bba45e064d8a5cd86acaab468a5bf3ff42af7a376f
Sep 11 16:17:01.002210 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:01.002185    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d5d9f0_49b4_4ac7_ab61_a3dd59301b65.slice/crio-9299ce2dd66aaed360069d551c535289f50d96eb13350e6d1a39ced17a99bf99 WatchSource:0}: Error finding container 9299ce2dd66aaed360069d551c535289f50d96eb13350e6d1a39ced17a99bf99: Status 404 returned error can't find the container with id 9299ce2dd66aaed360069d551c535289f50d96eb13350e6d1a39ced17a99bf99
Sep 11 16:17:01.004334 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:01.004266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04975c24_12e2_4cfc_a67e_f7b83f2ce37a.slice/crio-9b9ca2ebc80573f7b1ae8e9d51a25bea65d04f9a25abe22f5cb6484444be2bd8 WatchSource:0}: Error finding container 9b9ca2ebc80573f7b1ae8e9d51a25bea65d04f9a25abe22f5cb6484444be2bd8: Status 404 returned error can't find the container with id 9b9ca2ebc80573f7b1ae8e9d51a25bea65d04f9a25abe22f5cb6484444be2bd8
Sep 11 16:17:01.008370 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:01.007608    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88ca20e_3919_44b3_b0d7_18ae7a2f1c97.slice/crio-3e597caa5e3f962c393d19bbcd84ab470d1aeaedd96813a85d80e60158ac13e0 WatchSource:0}: Error finding container 3e597caa5e3f962c393d19bbcd84ab470d1aeaedd96813a85d80e60158ac13e0: Status 404 returned error can't find the container with id 3e597caa5e3f962c393d19bbcd84ab470d1aeaedd96813a85d80e60158ac13e0
Sep 11 16:17:01.010768 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:01.009638    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536a788b_d062_40b8_9243_8c171bd82b95.slice/crio-30550c558731bf728528b6736c0414bb3c8fd588500d531cb07f376ad5722e9f WatchSource:0}: Error finding container 30550c558731bf728528b6736c0414bb3c8fd588500d531cb07f376ad5722e9f: Status 404 returned error can't find the container with id 30550c558731bf728528b6736c0414bb3c8fd588500d531cb07f376ad5722e9f
Sep 11 16:17:01.036078 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.035927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:01.036197 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.036175    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:01.049009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.048980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hbsz8" event={"ID":"b9c3fd3d-faf5-4651-ba28-8b10cb47243c","Type":"ContainerStarted","Data":"3cc6578f2de6f81cc663600351bd03452904492194fab9d1add43ac2c5e505e6"}
Sep 11 16:17:01.050125 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.050100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"9b9ca2ebc80573f7b1ae8e9d51a25bea65d04f9a25abe22f5cb6484444be2bd8"}
Sep 11 16:17:01.051771 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.051751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f8rwn" event={"ID":"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65","Type":"ContainerStarted","Data":"9299ce2dd66aaed360069d551c535289f50d96eb13350e6d1a39ced17a99bf99"}
Sep 11 16:17:01.052719 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.052699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn" event={"ID":"536a788b-d062-40b8-9243-8c171bd82b95","Type":"ContainerStarted","Data":"30550c558731bf728528b6736c0414bb3c8fd588500d531cb07f376ad5722e9f"}
Sep 11 16:17:01.055997 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.055977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjnjb" event={"ID":"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97","Type":"ContainerStarted","Data":"3e597caa5e3f962c393d19bbcd84ab470d1aeaedd96813a85d80e60158ac13e0"}
Sep 11 16:17:01.058919 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.058890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5nf4" event={"ID":"72e00e16-c3b3-4c18-b68f-f491df7b9a8f","Type":"ContainerStarted","Data":"c5f4b806b776b9602bfeb5bba45e064d8a5cd86acaab468a5bf3ff42af7a376f"}
Sep 11 16:17:01.061118 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.061099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz7hh" event={"ID":"d6a89602-b607-47e9-bd5d-8c2a5daa01fb","Type":"ContainerStarted","Data":"19f997ff13146f1ec626de2f96c8c4a5efae2c26d6830c4759e374f767c4e5d3"}
Sep 11 16:17:01.062170 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.062145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjnxd" event={"ID":"a478cd40-13f0-40f6-8d19-12af35e82945","Type":"ContainerStarted","Data":"06767ca8c709c85237ba81f953a3d475f87e59873e72e938a1a3c9186ca3a025"}
Sep 11 16:17:01.063266 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.063245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerStarted","Data":"2240d65d1b616900ed15a690d62715d51da6a4e1193959a9b18903eac15fc7f0"}
Sep 11 16:17:01.539052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.538898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:01.539214 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.539064    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:01.539214 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.539155    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:03.539135415 +0000 UTC m=+6.033035222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:01.639586 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.639546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:01.639867 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.639603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:01.639867 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.639791    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:01.639867 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.639858    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:03.639837505 +0000 UTC m=+6.133737319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:01.640325 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.640298    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:01.640401 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.640326    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:01.640401 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.640340    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:01.640401 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:01.640383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:03.640369589 +0000 UTC m=+6.134269382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:01.960642 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.960553    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 16:11:59 +0000 UTC" deadline="2028-05-06 22:45:52.519007267 +0000 UTC"
Sep 11 16:17:01.960642 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:01.960594    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14478h28m50.55841846s"
Sep 11 16:17:02.038123 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.038092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:02.038306 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:02.038231    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:02.038384 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.038308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:02.038443 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:02.038379    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:02.078454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.077526    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a022f343c02e056acab4e9c92407e8c" containerID="32df3f7c032dce455d5d11a37833901637320af1f485470a81cfb2fc241ce745" exitCode=0
Sep 11 16:17:02.078454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.077637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerDied","Data":"32df3f7c032dce455d5d11a37833901637320af1f485470a81cfb2fc241ce745"}
Sep 11 16:17:02.102127 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.102094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" event={"ID":"37310a8518ae4b4c8f438099a305077e","Type":"ContainerStarted","Data":"fc34bd9a91378c98f3c20c0ab8ea4631ab49eb92d299d343106349900e98b66d"}
Sep 11 16:17:02.124692 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:02.124607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-144.ec2.internal" podStartSLOduration=3.124586642 podStartE2EDuration="3.124586642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:02.123963166 +0000 UTC m=+4.617862982" watchObservedRunningTime="2026-09-11 16:17:02.124586642 +0000 UTC m=+4.618486456"
Sep 11 16:17:03.036826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:03.036748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:03.037516 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.036882    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:03.118320 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:03.118285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" event={"ID":"0a022f343c02e056acab4e9c92407e8c","Type":"ContainerStarted","Data":"f172c59ac5695c2e918f48898c18cbc9d385a5a4d2e7e067f1b73f989868d298"}
Sep 11 16:17:03.558786 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:03.558151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:03.558786 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.558318    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:03.558786 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.558379    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.558362486 +0000 UTC m=+10.052262292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:03.658577 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:03.658534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:03.658783 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:03.658589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:03.658783 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.658722    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:03.658890 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.658788    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.658770748 +0000 UTC m=+10.152670543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:03.659235 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.659213    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:03.659306 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.659243    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:03.659306 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.659257    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:03.659306 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:03.659304    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:07.659289022 +0000 UTC m=+10.153188818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:04.036359 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:04.036282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:04.036519 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:04.036417    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:04.036896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:04.036874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:04.037286 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:04.036981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:05.035642 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:05.035579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:05.035940 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:05.035828    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:06.036188 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:06.036138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:06.036667 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:06.036190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:06.036667 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:06.036261    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:06.036667 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:06.036405    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:07.036023 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:07.035992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:07.036214 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.036094    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:07.593152 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:07.593013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:07.593348 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.593172    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.593348 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.593242    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.593223814 +0000 UTC m=+18.087123618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:07.694457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:07.694509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.694670    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.694751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.694732079 +0000 UTC m=+18.188631872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.695163    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.695182    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.695194    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:07.695273 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:07.695239    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:15.695224567 +0000 UTC m=+18.189124361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:08.036908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:08.036816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:08.036908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:08.036858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:08.037408 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:08.036957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:08.037408 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:08.037087    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:09.036619 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:09.036586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:09.036856 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:09.036747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:10.036111 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:10.036074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:10.036563 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:10.036265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:10.036563 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:10.036384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:10.036563 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:10.036261    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:11.036661 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:11.036626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:11.037094 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:11.036771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:12.036155 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:12.036075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:12.036321 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:12.036210    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:12.036321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:12.036266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:12.036433 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:12.036359    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:13.035986 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:13.035954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:13.036381 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:13.036067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:14.036331 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:14.036293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:14.036825 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:14.036345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:14.036825 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:14.036459    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:14.036825 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:14.036580    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:15.035715 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:15.035671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:15.035978 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.035791    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:15.654524 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:15.654487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:15.655006 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.654639    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:15.655006 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.654720    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:17:31.654700799 +0000 UTC m=+34.148600593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:15.754881 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:15.754844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:15.755064 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:15.754901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:15.755064 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.754998    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:15.755064 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.755026    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:15.755064 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.755040    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:15.755263 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.755007    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:15.755263 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.755100    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:31.755079394 +0000 UTC m=+34.248979190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:15.755263 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:15.755151    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:17:31.755130301 +0000 UTC m=+34.249030107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:16.036821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:16.036544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:16.036986 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:16.036603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:16.036986 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:16.036872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:16.036986 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:16.036934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:17.035856 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:17.035817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:17.036349 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:17.035955    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:18.036884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:18.036846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:18.037338 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:18.036968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:18.037338 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:18.037007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:18.037338 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:18.037097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:19.038821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.036089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:19.038821 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:19.036208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:19.152132 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.151763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn" event={"ID":"536a788b-d062-40b8-9243-8c171bd82b95","Type":"ContainerStarted","Data":"63ce66df1b2831df87a2c06fa63923ba516a31314996316dc79f0caeef90af9b"}
Sep 11 16:17:19.153800 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.153738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5nf4" event={"ID":"72e00e16-c3b3-4c18-b68f-f491df7b9a8f","Type":"ContainerStarted","Data":"5fb1ef1afdaa7b7d97513d40afe57c81af4928ee72c41a462e3fb11a2733e0b1"}
Sep 11 16:17:19.159287 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.158850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bz7hh" event={"ID":"d6a89602-b607-47e9-bd5d-8c2a5daa01fb","Type":"ContainerStarted","Data":"14d51b9e7f660f08fdeb2b1116a144655b8c3fef0b391b417166b9a752190336"}
Sep 11 16:17:19.161606 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.161494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjnxd" event={"ID":"a478cd40-13f0-40f6-8d19-12af35e82945","Type":"ContainerStarted","Data":"faf449db25e45d58f0e3e5979951ac47ef29a7e0f3c6bc19b73cb21b537850be"}
Sep 11 16:17:19.163292 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.163261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerStarted","Data":"de3baa236b8644c6322275cd18170382b5312c8142339473baf204674d8178f5"}
Sep 11 16:17:19.165637 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.165579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hbsz8" event={"ID":"b9c3fd3d-faf5-4651-ba28-8b10cb47243c","Type":"ContainerStarted","Data":"1be03a994b7042784b71c809f88296d3bc7ffefa22129b39c254b11a4cb72a3b"}
Sep 11 16:17:19.168144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.168113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"7116e3931eeb0f9ab3e78d621eb09ca1ce1c94cb2ab31ebdc24ce09f30f5402e"}
Sep 11 16:17:19.168245 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.168195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"7ee8d0dded985b8fd03bf208c37bee12bcda86c027d744d648e5b2e6cb342a32"}
Sep 11 16:17:19.170414 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.170391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f8rwn" event={"ID":"73d5d9f0-49b4-4ac7-ab61-a3dd59301b65","Type":"ContainerStarted","Data":"67b95a01ffd85db722fbd46294c6971ce38b422b8347fa415dd6e9ad1734cd21"}
Sep 11 16:17:19.174317 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.174274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g5nf4" podStartSLOduration=3.282779984 podStartE2EDuration="21.174261545s" totalImagesPullingTime="17.891481561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:19.174183457 +0000 UTC m=+21.668083269" watchObservedRunningTime="2026-09-11 16:17:19.174261545 +0000 UTC m=+21.668161361"
Sep 11 16:17:19.174425 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.174368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-144.ec2.internal" podStartSLOduration=20.174363914 podStartE2EDuration="20.174363914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:03.13541986 +0000 UTC m=+5.629319673" watchObservedRunningTime="2026-09-11 16:17:19.174363914 +0000 UTC m=+21.668263725"
Sep 11 16:17:19.192102 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.192018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bz7hh" podStartSLOduration=3.34429293 podStartE2EDuration="21.191996209s" totalImagesPullingTime="17.847703279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:19.191980547 +0000 UTC m=+21.685880359" watchObservedRunningTime="2026-09-11 16:17:19.191996209 +0000 UTC m=+21.685896021"
Sep 11 16:17:19.217455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.217362    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hbsz8" podStartSLOduration=3.25746689 podStartE2EDuration="21.217347073s" totalImagesPullingTime="17.959880183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:19.216796942 +0000 UTC m=+21.710696758" watchObservedRunningTime="2026-09-11 16:17:19.217347073 +0000 UTC m=+21.711246884"
Sep 11 16:17:19.281986 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.281942    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f8rwn" podStartSLOduration=3.437900121 podStartE2EDuration="21.281925334s" totalImagesPullingTime="17.844025213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:19.25636491 +0000 UTC m=+21.750264723" watchObservedRunningTime="2026-09-11 16:17:19.281925334 +0000 UTC m=+21.775825146"
Sep 11 16:17:19.282190 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:19.282163    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mjnxd" podStartSLOduration=3.409170947 podStartE2EDuration="21.282151906s" totalImagesPullingTime="17.872980959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:19.281508811 +0000 UTC m=+21.775408625" watchObservedRunningTime="2026-09-11 16:17:19.282151906 +0000 UTC m=+21.776051718"
Sep 11 16:17:20.036319 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.036287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:20.036536 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:20.036416    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:20.036536 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.036473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:20.036652 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:20.036600    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:20.175743 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.175706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"1fdafa82767b7a24d009740c5372c51b2ff628b58b9697684dc2b7d167f7b0f4"}
Sep 11 16:17:20.176519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.175795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"3b17b77649144112ae0f0d1afabc56d687b96d9caf9a62c241b4fba31bef5d10"}
Sep 11 16:17:20.176519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.175835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"82224bce273fc2c1dee9bf5b61da79dbd4e52db3a07288f9e30e5b87dc921d15"}
Sep 11 16:17:20.176519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.175863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"8920170c3f9d80ad302d063e9a61128751c4be44ff3db707d299a62d3725e098"}
Sep 11 16:17:20.177098 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.177077    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="de3baa236b8644c6322275cd18170382b5312c8142339473baf204674d8178f5" exitCode=0
Sep 11 16:17:20.177189 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.177167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"de3baa236b8644c6322275cd18170382b5312c8142339473baf204674d8178f5"}
Sep 11 16:17:20.325784 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:20.325751    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 16:17:21.036138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.036109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:21.036328 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:21.036226    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:21.123452 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.123347    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T16:17:20.325781315Z","UUID":"7730f9c1-6129-44c0-a23b-e279347f87d8","Handler":null,"Name":"","Endpoint":""}
Sep 11 16:17:21.125365 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.125178    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 16:17:21.125365 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.125211    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 16:17:21.181361 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.181317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rjnjb" event={"ID":"a88ca20e-3919-44b3-b0d7-18ae7a2f1c97","Type":"ContainerStarted","Data":"05547b63d88fd08d338f2ed19231e13bb863daa943847b21c22768852b301c53"}
Sep 11 16:17:21.186179 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.186143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn" event={"ID":"536a788b-d062-40b8-9243-8c171bd82b95","Type":"ContainerStarted","Data":"b75dad93dad3dd964ac5f3a43abd5d0d91094854830a67def44d3fcc12a82bb7"}
Sep 11 16:17:21.198996 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:21.198934    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rjnjb" podStartSLOduration=5.362113791 podStartE2EDuration="23.198914918s" totalImagesPullingTime="17.836801127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:21.19851574 +0000 UTC m=+23.692415553" watchObservedRunningTime="2026-09-11 16:17:21.198914918 +0000 UTC m=+23.692814733"
Sep 11 16:17:22.036621 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:22.036349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:22.036821 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:22.036747    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:22.036934 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:22.036349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:22.037071 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:22.037043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:22.191489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:22.191377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn" event={"ID":"536a788b-d062-40b8-9243-8c171bd82b95","Type":"ContainerStarted","Data":"73723277290d25eb3240a66e9b123f0f13b1d6c63d84e7bba36bcef5c7e6fda2"}
Sep 11 16:17:22.195419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:22.195391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"3a4786c9c0cb1373eccad37ac32b35af3684aa179c30f390af3c5ee18c7b46c5"}
Sep 11 16:17:22.213785 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:22.213730    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kfkvn" podStartSLOduration=4.037732651 podStartE2EDuration="24.213713128s" totalImagesPullingTime="20.175980477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:22.212352144 +0000 UTC m=+24.706251959" watchObservedRunningTime="2026-09-11 16:17:22.213713128 +0000 UTC m=+24.707612940"
Sep 11 16:17:23.036063 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:23.036031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:23.036272 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:23.036150    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:23.511074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:23.510992    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:23.511648 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:23.511626    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:24.036466 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:24.036433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:24.036638 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:24.036434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:24.036638 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:24.036560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:24.036802 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:24.036649    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:24.200775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:24.200690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:24.201231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:24.201207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f8rwn"
Sep 11 16:17:25.036074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:25.035845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:25.036424 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:25.036100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:25.204518 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:25.204479    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="58e310e8ad9886a0186427f8a11e9afd27b0c0cbabfffb0ccbda3dea7e80977e" exitCode=0
Sep 11 16:17:25.204787 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:25.204559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"58e310e8ad9886a0186427f8a11e9afd27b0c0cbabfffb0ccbda3dea7e80977e"}
Sep 11 16:17:25.208263 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:25.208236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" event={"ID":"04975c24-12e2-4cfc-a67e-f7b83f2ce37a","Type":"ContainerStarted","Data":"68e9ef1e590b133cc4e243f2a82ec9d74d1baf32890b92eac191113deacd2cc5"}
Sep 11 16:17:25.275557 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:25.275508    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc" podStartSLOduration=9.37192394 podStartE2EDuration="27.275488236s" totalImagesPullingTime="17.903564296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:25.274046597 +0000 UTC m=+27.767946409" watchObservedRunningTime="2026-09-11 16:17:25.275488236 +0000 UTC m=+27.769388048"
Sep 11 16:17:26.036556 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.036522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:26.037120 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.036522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:26.037120 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:26.036667    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:26.037120 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:26.036715    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:26.211257 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.211230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:26.211356 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.211318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:26.211396 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.211374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:26.228627 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.228603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:26.229314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.229292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:17:26.632350 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.632273    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tb8lh"]
Sep 11 16:17:26.632510 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.632437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:26.632575 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:26.632545    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:26.635640 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.635617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjf55"]
Sep 11 16:17:26.635779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.635727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:26.635822 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:26.635803    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:26.652426 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.652394    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-prnmx"]
Sep 11 16:17:26.652532 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:26.652488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:26.652583 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:26.652567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:27.216970 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:27.216936    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="479a6ca84fa20ac0d59d368202a42028a59a197d8c2d22ad85ee31d15b47accb" exitCode=0
Sep 11 16:17:27.217659 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:27.217028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"479a6ca84fa20ac0d59d368202a42028a59a197d8c2d22ad85ee31d15b47accb"}
Sep 11 16:17:28.036823 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:28.036785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:28.036957 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:28.036884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:28.036957 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:28.036910    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:28.037099 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:28.036971    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:29.036012 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:29.035778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:29.036393 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:29.036036    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:29.223584 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:29.223550    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="9d225cd9395aaa6c19abd5a4254b8653e9af7143498e004ed180c83e781c9d06" exitCode=0
Sep 11 16:17:29.223764 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:29.223611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"9d225cd9395aaa6c19abd5a4254b8653e9af7143498e004ed180c83e781c9d06"}
Sep 11 16:17:30.036522 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:30.036482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:30.036989 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:30.036482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:30.036989 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:30.036710    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:30.036989 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:30.036766    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:31.036500 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:31.036464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:31.036657 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.036587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-prnmx" podUID="ba0fbdc5-0204-4bd7-aa2a-d409b740646c"
Sep 11 16:17:31.670240 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:31.670198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:31.670524 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.670328    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:31.670524 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.670408    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.670388529 +0000 UTC m=+66.164288323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 16:17:31.771029 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:31.770984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:31.771029 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:31.771030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771137    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771154    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771182    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771195    2563 projected.go:196] Error preparing data for projected volume kube-api-access-744mj for pod openshift-network-diagnostics/network-check-target-tb8lh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771204    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret podName:ba0fbdc5-0204-4bd7-aa2a-d409b740646c nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.771186496 +0000 UTC m=+66.265086292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret") pod "global-pull-secret-syncer-prnmx" (UID: "ba0fbdc5-0204-4bd7-aa2a-d409b740646c") : object "kube-system"/"original-pull-secret" not registered
Sep 11 16:17:31.771250 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:31.771234    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj podName:f3a689a5-adf0-43f6-9108-1ad0a6ee67f2 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:03.77122256 +0000 UTC m=+66.265122363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-744mj" (UniqueName: "kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj") pod "network-check-target-tb8lh" (UID: "f3a689a5-adf0-43f6-9108-1ad0a6ee67f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 16:17:32.036441 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.036405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:32.036627 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.036417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:32.036627 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:32.036566    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zjf55" podUID="13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f"
Sep 11 16:17:32.036627 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:32.036591    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tb8lh" podUID="f3a689a5-adf0-43f6-9108-1ad0a6ee67f2"
Sep 11 16:17:32.811311 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.811280    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-144.ec2.internal" event="NodeReady"
Sep 11 16:17:32.811494 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.811418    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 16:17:32.865900 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.865864    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6hj5n"]
Sep 11 16:17:32.871212 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.871184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6qr6w"]
Sep 11 16:17:32.871356 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.871335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:32.874039 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.874015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:17:32.874183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.874160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 16:17:32.874484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.874463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:32.874900 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.874843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 16:17:32.877485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.877467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 16:17:32.877577 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.877525    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 16:17:32.878294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.878115    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 16:17:32.878457 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.878347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qbh4n\""
Sep 11 16:17:32.883027 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.883005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hj5n"]
Sep 11 16:17:32.889380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.889359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qr6w"]
Sep 11 16:17:32.979990 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.979950    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:32.980173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.980018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/38a822e7-2ba9-4d1e-b494-1056a88c7af8-kube-api-access-mctxp\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:32.980173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.980038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2d5abe-4939-4d13-97ac-78ba4b84d293-cert\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:32.980173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.980061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkn7\" (UniqueName: \"kubernetes.io/projected/7d2d5abe-4939-4d13-97ac-78ba4b84d293-kube-api-access-ztkn7\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:32.980173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.980112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e7-2ba9-4d1e-b494-1056a88c7af8-config-volume\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:32.980340 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:32.980205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a822e7-2ba9-4d1e-b494-1056a88c7af8-tmp-dir\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.036250 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.036209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:17:33.041852 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.041827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:17:33.081102 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.080997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.081102 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/38a822e7-2ba9-4d1e-b494-1056a88c7af8-kube-api-access-mctxp\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2d5abe-4939-4d13-97ac-78ba4b84d293-cert\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkn7\" (UniqueName: \"kubernetes.io/projected/7d2d5abe-4939-4d13-97ac-78ba4b84d293-kube-api-access-ztkn7\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:33.081134    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e7-2ba9-4d1e-b494-1056a88c7af8-config-volume\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a822e7-2ba9-4d1e-b494-1056a88c7af8-tmp-dir\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.081314 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:33.081219    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:33.581198485 +0000 UTC m=+36.075098276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:33.081610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38a822e7-2ba9-4d1e-b494-1056a88c7af8-tmp-dir\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.081937 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.081912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/38a822e7-2ba9-4d1e-b494-1056a88c7af8-config-volume\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.086372 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.086348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d2d5abe-4939-4d13-97ac-78ba4b84d293-cert\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:33.092054 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.092032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkn7\" (UniqueName: \"kubernetes.io/projected/7d2d5abe-4939-4d13-97ac-78ba4b84d293-kube-api-access-ztkn7\") pod \"ingress-canary-6qr6w\" (UID: \"7d2d5abe-4939-4d13-97ac-78ba4b84d293\") " pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:33.092054 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.092041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctxp\" (UniqueName: \"kubernetes.io/projected/38a822e7-2ba9-4d1e-b494-1056a88c7af8-kube-api-access-mctxp\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.190837 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.190801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qr6w"
Sep 11 16:17:33.584671 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.584521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:33.584788 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:33.584639    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:33.584828 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:33.584795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:34.584772933 +0000 UTC m=+37.078672736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:33.711333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:33.711303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qr6w"]
Sep 11 16:17:33.714508 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:17:33.714471    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2d5abe_4939_4d13_97ac_78ba4b84d293.slice/crio-b9e65523b4bf9345ae824d9294cf457eedfb50cc0226d0707d2440536a8ab5a9 WatchSource:0}: Error finding container b9e65523b4bf9345ae824d9294cf457eedfb50cc0226d0707d2440536a8ab5a9: Status 404 returned error can't find the container with id b9e65523b4bf9345ae824d9294cf457eedfb50cc0226d0707d2440536a8ab5a9
Sep 11 16:17:34.036420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.036380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:17:34.036605 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.036586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:17:34.039370 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.039349    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:17:34.040423 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.040403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:17:34.040423 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.040411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:17:34.040558 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.040408    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:17:34.040558 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.040406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:17:34.242950 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.242912    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="84a050d1749dbdfebd8dfad5be75f481a3a9535bf7df5e217db6b9bf37caf92a" exitCode=0
Sep 11 16:17:34.243496 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.242995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"84a050d1749dbdfebd8dfad5be75f481a3a9535bf7df5e217db6b9bf37caf92a"}
Sep 11 16:17:34.244559 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.244535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qr6w" event={"ID":"7d2d5abe-4939-4d13-97ac-78ba4b84d293","Type":"ContainerStarted","Data":"b9e65523b4bf9345ae824d9294cf457eedfb50cc0226d0707d2440536a8ab5a9"}
Sep 11 16:17:34.591639 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:34.591541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:34.591823 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:34.591718    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:34.591823 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:34.591790    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:36.591771519 +0000 UTC m=+39.085671331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:35.250689 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:35.250639    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ede9424-cc8b-4a73-b30f-8bcf98e8e129" containerID="ffcac68db29e2805f978905f14a511570b0d9b78952c932f768edd136310542f" exitCode=0
Sep 11 16:17:35.251310 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:35.250701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerDied","Data":"ffcac68db29e2805f978905f14a511570b0d9b78952c932f768edd136310542f"}
Sep 11 16:17:36.255618 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:36.255592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rx84r" event={"ID":"0ede9424-cc8b-4a73-b30f-8bcf98e8e129","Type":"ContainerStarted","Data":"bc5ddbc6263b4c76e700f347a642eded36885aeaba07cc255ccdb4367329df1f"}
Sep 11 16:17:36.302380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:36.302323    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rx84r" podStartSLOduration=9.479672006 podStartE2EDuration="38.302299206s" totalImagesPullingTime="28.8226272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:36.30149144 +0000 UTC m=+38.795391249" watchObservedRunningTime="2026-09-11 16:17:36.302299206 +0000 UTC m=+38.796199019"
Sep 11 16:17:36.607070 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:36.606976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:36.607224 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:36.607116    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:36.607224 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:36.607174    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:40.60715979 +0000 UTC m=+43.101059585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:37.259779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:37.259741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qr6w" event={"ID":"7d2d5abe-4939-4d13-97ac-78ba4b84d293","Type":"ContainerStarted","Data":"e43fdc9325cdc67eefc120b8c7eda9658932bda2bb25b30ce38cf9adef1868e3"}
Sep 11 16:17:37.277570 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:37.277522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6qr6w" podStartSLOduration=2.7476543490000003 podStartE2EDuration="5.277504431s" totalImagesPullingTime="2.529850082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:37.277029409 +0000 UTC m=+39.770929222" watchObservedRunningTime="2026-09-11 16:17:37.277504431 +0000 UTC m=+39.771404234"
Sep 11 16:17:40.634772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:40.634727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:40.635171 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:40.634890    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:40.635171 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:40.634956    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:17:48.634939995 +0000 UTC m=+51.128839784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:48.690348 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:48.690302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:17:48.690799 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:48.690417    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:17:48.690799 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:17:48.690478    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:04.690464549 +0000 UTC m=+67.184364338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:17:58.229419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:17:58.229388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shkzc"
Sep 11 16:18:03.692083 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.692042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:18:03.697511 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.697487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 16:18:03.702794 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:03.702779    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 16:18:03.702854 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:03.702838    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs podName:13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f nodeName:}" failed. No retries permitted until 2026-09-11 16:19:07.702821326 +0000 UTC m=+130.196721116 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs") pod "network-metrics-daemon-zjf55" (UID: "13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f") : secret "metrics-daemon-secret" not found
Sep 11 16:18:03.792761 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.792727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:18:03.792828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.792765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:18:03.795792 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.795770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 16:18:03.795862 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.795772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 16:18:03.805954 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.805928    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 16:18:03.806164 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.806144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ba0fbdc5-0204-4bd7-aa2a-d409b740646c-original-pull-secret\") pod \"global-pull-secret-syncer-prnmx\" (UID: \"ba0fbdc5-0204-4bd7-aa2a-d409b740646c\") " pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:18:03.816364 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.816342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-744mj\" (UniqueName: \"kubernetes.io/projected/f3a689a5-adf0-43f6-9108-1ad0a6ee67f2-kube-api-access-744mj\") pod \"network-check-target-tb8lh\" (UID: \"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2\") " pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:18:03.946368 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:03.946288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-prnmx"
Sep 11 16:18:04.053484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.053458    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrmtk\""
Sep 11 16:18:04.060626 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.060610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:18:04.244756 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.244655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-prnmx"]
Sep 11 16:18:04.249773 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:04.249747    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0fbdc5_0204_4bd7_aa2a_d409b740646c.slice/crio-f8b7cd5ec22113bde39ecbb7e391ed1ccc0e21fc690ca67fd3473e3c0b77309a WatchSource:0}: Error finding container f8b7cd5ec22113bde39ecbb7e391ed1ccc0e21fc690ca67fd3473e3c0b77309a: Status 404 returned error can't find the container with id f8b7cd5ec22113bde39ecbb7e391ed1ccc0e21fc690ca67fd3473e3c0b77309a
Sep 11 16:18:04.325129 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.325095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-prnmx" event={"ID":"ba0fbdc5-0204-4bd7-aa2a-d409b740646c","Type":"ContainerStarted","Data":"f8b7cd5ec22113bde39ecbb7e391ed1ccc0e21fc690ca67fd3473e3c0b77309a"}
Sep 11 16:18:04.346313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.346285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tb8lh"]
Sep 11 16:18:04.349460 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:04.349435    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a689a5_adf0_43f6_9108_1ad0a6ee67f2.slice/crio-7f79bf37091fa25c0b0dcc5fd50218e664b664f98335fb6ea26846d9d11872eb WatchSource:0}: Error finding container 7f79bf37091fa25c0b0dcc5fd50218e664b664f98335fb6ea26846d9d11872eb: Status 404 returned error can't find the container with id 7f79bf37091fa25c0b0dcc5fd50218e664b664f98335fb6ea26846d9d11872eb
Sep 11 16:18:04.698827 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:04.698789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:18:04.699202 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:04.698954    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:18:04.699202 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:04.699029    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:36.699009709 +0000 UTC m=+99.192909516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:05.330295 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:05.330254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tb8lh" event={"ID":"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2","Type":"ContainerStarted","Data":"7f79bf37091fa25c0b0dcc5fd50218e664b664f98335fb6ea26846d9d11872eb"}
Sep 11 16:18:09.341748 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:09.341709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tb8lh" event={"ID":"f3a689a5-adf0-43f6-9108-1ad0a6ee67f2","Type":"ContainerStarted","Data":"10387f1627e56120a5393a4fc4013212b8da4477055d26f80e40471ab34802d6"}
Sep 11 16:18:09.342184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:09.341799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:18:09.343180 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:09.343156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-prnmx" event={"ID":"ba0fbdc5-0204-4bd7-aa2a-d409b740646c","Type":"ContainerStarted","Data":"818bc0ebc57df17232ea3accd6dbe3fdd39315502b2e12b7a7185cc6013c9497"}
Sep 11 16:18:09.359692 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:09.359638    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tb8lh" podStartSLOduration=66.919498918 podStartE2EDuration="1m11.359625239s" totalImagesPullingTime="4.440126321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:09.358601587 +0000 UTC m=+71.852501398" watchObservedRunningTime="2026-09-11 16:18:09.359625239 +0000 UTC m=+71.853525045"
Sep 11 16:18:09.376911 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:09.376862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-prnmx" podStartSLOduration=66.833217856 podStartE2EDuration="1m11.376851326s" totalImagesPullingTime="4.54363347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:09.376248498 +0000 UTC m=+71.870148309" watchObservedRunningTime="2026-09-11 16:18:09.376851326 +0000 UTC m=+71.870751137"
Sep 11 16:18:19.486409 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.486367    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-8drrk"]
Sep 11 16:18:19.490404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.490380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.497555 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.497535    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 16:18:19.497555 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.497547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-q5dhp\""
Sep 11 16:18:19.498876 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.498859    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 16:18:19.498964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.498889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 16:18:19.498964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.498898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 16:18:19.503845 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.503826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 16:18:19.506983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.506964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-8drrk"]
Sep 11 16:18:19.600322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-service-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.600322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600328    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-snapshots\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.600599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvlj\" (UniqueName: \"kubernetes.io/projected/4de17a84-cf83-4cee-a71b-d70c7cfba916-kube-api-access-7cvlj\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.600599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4de17a84-cf83-4cee-a71b-d70c7cfba916-serving-cert\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.600599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-tmp\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.600599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.600571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvlj\" (UniqueName: \"kubernetes.io/projected/4de17a84-cf83-4cee-a71b-d70c7cfba916-kube-api-access-7cvlj\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4de17a84-cf83-4cee-a71b-d70c7cfba916-serving-cert\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-tmp\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-service-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.701858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.701702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-snapshots\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.702320 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.702290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-tmp\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.702449 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.702323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-service-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.702449 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.702421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4de17a84-cf83-4cee-a71b-d70c7cfba916-snapshots\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.702565 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.702548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4de17a84-cf83-4cee-a71b-d70c7cfba916-trusted-ca-bundle\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.703984 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.703968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4de17a84-cf83-4cee-a71b-d70c7cfba916-serving-cert\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.716155 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.716134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvlj\" (UniqueName: \"kubernetes.io/projected/4de17a84-cf83-4cee-a71b-d70c7cfba916-kube-api-access-7cvlj\") pod \"insights-operator-5844b856db-8drrk\" (UID: \"4de17a84-cf83-4cee-a71b-d70c7cfba916\") " pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:19.798274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:19.798185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-8drrk"
Sep 11 16:18:20.073233 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:20.073159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-8drrk"]
Sep 11 16:18:20.076009 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:20.075981    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de17a84_cf83_4cee_a71b_d70c7cfba916.slice/crio-36755d688ae0ccc9d01c110c952f117fd275215858ee5a4bd91ba3b67316f910 WatchSource:0}: Error finding container 36755d688ae0ccc9d01c110c952f117fd275215858ee5a4bd91ba3b67316f910: Status 404 returned error can't find the container with id 36755d688ae0ccc9d01c110c952f117fd275215858ee5a4bd91ba3b67316f910
Sep 11 16:18:20.370373 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:20.370229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8drrk" event={"ID":"4de17a84-cf83-4cee-a71b-d70c7cfba916","Type":"ContainerStarted","Data":"36755d688ae0ccc9d01c110c952f117fd275215858ee5a4bd91ba3b67316f910"}
Sep 11 16:18:22.380743 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:22.380616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8drrk" event={"ID":"4de17a84-cf83-4cee-a71b-d70c7cfba916","Type":"ContainerStarted","Data":"45cff109c84307326c138ca2364fcba53f8f71866b886121357917568270c854"}
Sep 11 16:18:22.401596 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:22.401545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-8drrk" podStartSLOduration=1.4498750120000001 podStartE2EDuration="3.401510347s" totalImagesPullingTime="1.951635335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:22.399746807 +0000 UTC m=+84.893646622" watchObservedRunningTime="2026-09-11 16:18:22.401510347 +0000 UTC m=+84.895410158"
Sep 11 16:18:23.546206 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.546155    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h6lpd"]
Sep 11 16:18:23.549443 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.549420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.552641 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.552598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 16:18:23.552786 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.552671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 16:18:23.554565 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.554546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26bj7\""
Sep 11 16:18:23.568061 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.568033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6lpd"]
Sep 11 16:18:23.630300 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.630262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.630473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.630305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.630473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.630353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-crio-socket\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.630473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.630373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrl6w\" (UniqueName: \"kubernetes.io/projected/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-api-access-vrl6w\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.630473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.630392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-data-volume\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-crio-socket\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731543 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-crio-socket\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731543 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:23.731409    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.731543 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrl6w\" (UniqueName: \"kubernetes.io/projected/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-api-access-vrl6w\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731543 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:23.731490    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.231473614 +0000 UTC m=+86.725373427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:23.731543 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-data-volume\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731732 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.731732 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:23.731663    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.731732 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:23.731718    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:24.231706137 +0000 UTC m=+86.725605942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:23.731912 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.731894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-data-volume\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:23.739765 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:23.739747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrl6w\" (UniqueName: \"kubernetes.io/projected/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-api-access-vrl6w\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:24.235170 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:24.235132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:24.235170 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:24.235170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:24.235377 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:24.235256    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.235377 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:24.235289    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:24.235377 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:24.235330    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.235310823 +0000 UTC m=+87.729210633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:24.235377 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:24.235345    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:25.235337958 +0000 UTC m=+87.729237752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.242387 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:25.242355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:25.242387 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:25.242388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:25.242805 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:25.242516    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.242805 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:25.242537    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.242805 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:25.242582    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.242568322 +0000 UTC m=+89.736468111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:25.242805 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:25.242597    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:27.242590255 +0000 UTC m=+89.736490045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:25.461362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:25.461336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5nf4_72e00e16-c3b3-4c18-b68f-f491df7b9a8f/dns-node-resolver/0.log"
Sep 11 16:18:26.461124 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:26.461097    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjnxd_a478cd40-13f0-40f6-8d19-12af35e82945/node-ca/0.log"
Sep 11 16:18:26.867039 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:26.866952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qr6w_7d2d5abe-4939-4d13-97ac-78ba4b84d293/serve-healthcheck-canary/0.log"
Sep 11 16:18:27.257356 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:27.257320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:27.257528 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:27.257362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:27.257528 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:27.257444    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.257528 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:27.257486    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:27.257644 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:27.257498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.257484509 +0000 UTC m=+93.751384299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:27.257644 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:27.257562    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.257546787 +0000 UTC m=+93.751446580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:29.545734 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.545689    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"]
Sep 11 16:18:29.548518 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.548501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.551240 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.551218    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bcx4d\""
Sep 11 16:18:29.551448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.551433    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:29.552243 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.552227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 16:18:29.552243 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.552238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:29.554886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.554868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 16:18:29.561085 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.561064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"]
Sep 11 16:18:29.575449 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.575423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvtv\" (UniqueName: \"kubernetes.io/projected/f9678d83-ed8e-41f3-9562-1f562984b9a2-kube-api-access-5tvtv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.575544 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.575487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9678d83-ed8e-41f3-9562-1f562984b9a2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.575584 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.575556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9678d83-ed8e-41f3-9562-1f562984b9a2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.650609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.650582    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"]
Sep 11 16:18:29.653415 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.653394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"]
Sep 11 16:18:29.653538 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.653515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.656830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.656796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.658052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.658034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x568w\""
Sep 11 16:18:29.658183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.658057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-t629d"]
Sep 11 16:18:29.658183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.658068    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:29.658272 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.658037    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:29.658272 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.658264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 16:18:29.659524 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.659436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 16:18:29.659524 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.659451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 16:18:29.659760 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.659741    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:29.659823 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.659808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gpsrt\""
Sep 11 16:18:29.659901 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.659885    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:29.661471 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.661452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.665127 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.665107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 16:18:29.665246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.665124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 16:18:29.665369 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.665352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:18:29.665435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.665373    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 16:18:29.665487 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.665472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vx6hw\""
Sep 11 16:18:29.673706 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.673668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"]
Sep 11 16:18:29.673839 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.673768    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 16:18:29.675960 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.675938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"]
Sep 11 16:18:29.676208 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.676181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9678d83-ed8e-41f3-9562-1f562984b9a2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.676293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.676239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9678d83-ed8e-41f3-9562-1f562984b9a2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.676293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.676270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.676391 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.676335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sx9\" (UniqueName: \"kubernetes.io/projected/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-kube-api-access-m4sx9\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.676391 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.676378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvtv\" (UniqueName: \"kubernetes.io/projected/f9678d83-ed8e-41f3-9562-1f562984b9a2-kube-api-access-5tvtv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.677365 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.677341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9678d83-ed8e-41f3-9562-1f562984b9a2-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.677940 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.677920    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-t629d"]
Sep 11 16:18:29.678822 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.678802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9678d83-ed8e-41f3-9562-1f562984b9a2-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.689076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.689048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvtv\" (UniqueName: \"kubernetes.io/projected/f9678d83-ed8e-41f3-9562-1f562984b9a2-kube-api-access-5tvtv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-6fq7n\" (UID: \"f9678d83-ed8e-41f3-9562-1f562984b9a2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.777352 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-config\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.777519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sx9\" (UniqueName: \"kubernetes.io/projected/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-kube-api-access-m4sx9\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.777519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-trusted-ca\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.777519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hds\" (UniqueName: \"kubernetes.io/projected/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-kube-api-access-22hds\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.777519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-config\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.777519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-serving-cert\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.777781 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83bfe23b-dd43-457f-90f2-377a82c3d309-serving-cert\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.777781 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2xx\" (UniqueName: \"kubernetes.io/projected/83bfe23b-dd43-457f-90f2-377a82c3d309-kube-api-access-tn2xx\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.777781 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.777720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.777890 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:29.777840    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:18:29.777923 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:29.777904    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls podName:91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:30.277887372 +0000 UTC m=+92.771787170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s7zqk" (UID: "91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4") : secret "samples-operator-tls" not found
Sep 11 16:18:29.788634 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.788605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sx9\" (UniqueName: \"kubernetes.io/projected/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-kube-api-access-m4sx9\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:29.856767 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.856643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"
Sep 11 16:18:29.878535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-trusted-ca\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.878662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hds\" (UniqueName: \"kubernetes.io/projected/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-kube-api-access-22hds\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.878662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-config\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.878662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-serving-cert\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.878662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83bfe23b-dd43-457f-90f2-377a82c3d309-serving-cert\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.878662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2xx\" (UniqueName: \"kubernetes.io/projected/83bfe23b-dd43-457f-90f2-377a82c3d309-kube-api-access-tn2xx\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.878976 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.878857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-config\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.879404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.879375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-config\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.879515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.879431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-config\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.879515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.879437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83bfe23b-dd43-457f-90f2-377a82c3d309-trusted-ca\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.880921 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.880897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-serving-cert\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.881005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.880975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83bfe23b-dd43-457f-90f2-377a82c3d309-serving-cert\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.890378 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.890354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hds\" (UniqueName: \"kubernetes.io/projected/0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a-kube-api-access-22hds\") pod \"service-ca-operator-cf7c776b7-l6qv5\" (UID: \"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.890489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.890460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2xx\" (UniqueName: \"kubernetes.io/projected/83bfe23b-dd43-457f-90f2-377a82c3d309-kube-api-access-tn2xx\") pod \"console-operator-696884fb9c-t629d\" (UID: \"83bfe23b-dd43-457f-90f2-377a82c3d309\") " pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:29.971930 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.971893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"
Sep 11 16:18:29.979137 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:29.979110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:30.128008 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.127935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n"]
Sep 11 16:18:30.130948 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:30.130919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9678d83_ed8e_41f3_9562_1f562984b9a2.slice/crio-dceb0993b6bc2a6a36979f1b71a0047a9c31b80891361c98cb4463e7c04b9e64 WatchSource:0}: Error finding container dceb0993b6bc2a6a36979f1b71a0047a9c31b80891361c98cb4463e7c04b9e64: Status 404 returned error can't find the container with id dceb0993b6bc2a6a36979f1b71a0047a9c31b80891361c98cb4463e7c04b9e64
Sep 11 16:18:30.247190 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.247157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5"]
Sep 11 16:18:30.255513 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:30.255488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b30d5ea_db78_4b29_b0a3_b5c4954ffd2a.slice/crio-4e9bb8ae4b5666a22fd8f971f2e6db5ff2bae35abb559b81dd3b24ee2efbe6f4 WatchSource:0}: Error finding container 4e9bb8ae4b5666a22fd8f971f2e6db5ff2bae35abb559b81dd3b24ee2efbe6f4: Status 404 returned error can't find the container with id 4e9bb8ae4b5666a22fd8f971f2e6db5ff2bae35abb559b81dd3b24ee2efbe6f4
Sep 11 16:18:30.266749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.266724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-t629d"]
Sep 11 16:18:30.269836 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:30.269813    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bfe23b_dd43_457f_90f2_377a82c3d309.slice/crio-b455533e8a63a7eb653d8df28d11d6b876736a8324a14cf6c3e36701e10a6613 WatchSource:0}: Error finding container b455533e8a63a7eb653d8df28d11d6b876736a8324a14cf6c3e36701e10a6613: Status 404 returned error can't find the container with id b455533e8a63a7eb653d8df28d11d6b876736a8324a14cf6c3e36701e10a6613
Sep 11 16:18:30.281631 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.281609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:30.281772 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:30.281758    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:18:30.281820 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:30.281810    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls podName:91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:31.281796236 +0000 UTC m=+93.775696027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s7zqk" (UID: "91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4") : secret "samples-operator-tls" not found
Sep 11 16:18:30.402142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.402047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5" event={"ID":"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a","Type":"ContainerStarted","Data":"4e9bb8ae4b5666a22fd8f971f2e6db5ff2bae35abb559b81dd3b24ee2efbe6f4"}
Sep 11 16:18:30.403165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.403139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n" event={"ID":"f9678d83-ed8e-41f3-9562-1f562984b9a2","Type":"ContainerStarted","Data":"dceb0993b6bc2a6a36979f1b71a0047a9c31b80891361c98cb4463e7c04b9e64"}
Sep 11 16:18:30.404142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:30.404113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-t629d" event={"ID":"83bfe23b-dd43-457f-90f2-377a82c3d309","Type":"ContainerStarted","Data":"b455533e8a63a7eb653d8df28d11d6b876736a8324a14cf6c3e36701e10a6613"}
Sep 11 16:18:31.289058 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:31.289017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:31.289058 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:31.289066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:31.289129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289197    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289217    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289271    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289284    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls podName:91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:33.289264293 +0000 UTC m=+95.783164099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s7zqk" (UID: "91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4") : secret "samples-operator-tls" not found
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289321    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.289305062 +0000 UTC m=+101.783204866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:31.289579 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:31.289338    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:39.289328963 +0000 UTC m=+101.783228760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : configmap "kube-rbac-proxy" not found
Sep 11 16:18:33.307043 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.306954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:33.307399 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:33.307130    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:18:33.307399 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:33.307218    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls podName:91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:37.30719705 +0000 UTC m=+99.801096860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s7zqk" (UID: "91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4") : secret "samples-operator-tls" not found
Sep 11 16:18:33.416465 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.416426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n" event={"ID":"f9678d83-ed8e-41f3-9562-1f562984b9a2","Type":"ContainerStarted","Data":"332bfa05688384cb41e61e3b562afe4122f40bebc7aa19580b8aceaeaaf3ff14"}
Sep 11 16:18:33.418125 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.418103    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/0.log"
Sep 11 16:18:33.418258 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.418140    2563 generic.go:356] "Generic (PLEG): container finished" podID="83bfe23b-dd43-457f-90f2-377a82c3d309" containerID="1216d88e83c29d869d03c45b17c76b8f714730a6c856b8f0ff8f5596020cef1e" exitCode=255
Sep 11 16:18:33.418258 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.418204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-t629d" event={"ID":"83bfe23b-dd43-457f-90f2-377a82c3d309","Type":"ContainerDied","Data":"1216d88e83c29d869d03c45b17c76b8f714730a6c856b8f0ff8f5596020cef1e"}
Sep 11 16:18:33.418440 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.418423    2563 scope.go:117] "RemoveContainer" containerID="1216d88e83c29d869d03c45b17c76b8f714730a6c856b8f0ff8f5596020cef1e"
Sep 11 16:18:33.419879 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.419853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5" event={"ID":"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a","Type":"ContainerStarted","Data":"ed9727316974231ee999f603d13ab7a89381835acf65daec02bbf25efafb59af"}
Sep 11 16:18:33.441207 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.441157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n" podStartSLOduration=1.5781186759999999 podStartE2EDuration="4.441141512s" totalImagesPullingTime="2.863022836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:33.439878583 +0000 UTC m=+95.933778396" watchObservedRunningTime="2026-09-11 16:18:33.441141512 +0000 UTC m=+95.935041324"
Sep 11 16:18:33.458595 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:33.458544    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5" podStartSLOduration=1.712921127 podStartE2EDuration="4.458523395s" totalImagesPullingTime="2.745602268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:33.458055084 +0000 UTC m=+95.951954897" watchObservedRunningTime="2026-09-11 16:18:33.458523395 +0000 UTC m=+95.952423209"
Sep 11 16:18:34.424421 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.424352    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/1.log"
Sep 11 16:18:34.424846 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.424822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/0.log"
Sep 11 16:18:34.424901 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.424860    2563 generic.go:356] "Generic (PLEG): container finished" podID="83bfe23b-dd43-457f-90f2-377a82c3d309" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242" exitCode=255
Sep 11 16:18:34.424948 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.424934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-t629d" event={"ID":"83bfe23b-dd43-457f-90f2-377a82c3d309","Type":"ContainerDied","Data":"f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"}
Sep 11 16:18:34.424998 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.424979    2563 scope.go:117] "RemoveContainer" containerID="1216d88e83c29d869d03c45b17c76b8f714730a6c856b8f0ff8f5596020cef1e"
Sep 11 16:18:34.425183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.425161    2563 scope.go:117] "RemoveContainer" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"
Sep 11 16:18:34.425411 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:34.425391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:18:34.816152 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.816120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"]
Sep 11 16:18:34.820230 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.820213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:34.826304 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.826284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 16:18:34.826419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.826283    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 16:18:34.826419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.826319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 16:18:34.827001 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.826987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hnwcj\""
Sep 11 16:18:34.827063 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.827010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 16:18:34.842469 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.842439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"]
Sep 11 16:18:34.923631 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.923597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:34.923803 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:34.923639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpxs\" (UniqueName: \"kubernetes.io/projected/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-kube-api-access-9gpxs\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.024472 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.024429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.024472 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.024477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpxs\" (UniqueName: \"kubernetes.io/projected/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-kube-api-access-9gpxs\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.026975 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.026949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.033197 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.033158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpxs\" (UniqueName: \"kubernetes.io/projected/88987724-bcf8-49cf-ad6c-45bc2eed9c0d-kube-api-access-9gpxs\") pod \"managed-serviceaccount-addon-agent-57bb67df64-nkcjp\" (UID: \"88987724-bcf8-49cf-ad6c-45bc2eed9c0d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.135738 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.135620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"
Sep 11 16:18:35.413021 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.411407    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp"]
Sep 11 16:18:35.417000 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:35.416965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88987724_bcf8_49cf_ad6c_45bc2eed9c0d.slice/crio-907f3651ed32150df21335dfbf19285bf0ca01d88ce8a725d935fccee03de08b WatchSource:0}: Error finding container 907f3651ed32150df21335dfbf19285bf0ca01d88ce8a725d935fccee03de08b: Status 404 returned error can't find the container with id 907f3651ed32150df21335dfbf19285bf0ca01d88ce8a725d935fccee03de08b
Sep 11 16:18:35.429444 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.429426    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/1.log"
Sep 11 16:18:35.429780 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.429763    2563 scope.go:117] "RemoveContainer" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"
Sep 11 16:18:35.429948 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:35.429927    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:18:35.430635 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:35.430615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp" event={"ID":"88987724-bcf8-49cf-ad6c-45bc2eed9c0d","Type":"ContainerStarted","Data":"907f3651ed32150df21335dfbf19285bf0ca01d88ce8a725d935fccee03de08b"}
Sep 11 16:18:36.740606 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:36.740570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:18:36.741030 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:36.740764    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 16:18:36.741030 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:36.740845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls podName:38a822e7-2ba9-4d1e-b494-1056a88c7af8 nodeName:}" failed. No retries permitted until 2026-09-11 16:19:40.740824724 +0000 UTC m=+163.234724532 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls") pod "dns-default-6hj5n" (UID: "38a822e7-2ba9-4d1e-b494-1056a88c7af8") : secret "dns-default-metrics-tls" not found
Sep 11 16:18:37.345880 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:37.345838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:37.346055 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:37.345966    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 16:18:37.346055 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:37.346053    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls podName:91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:45.346021444 +0000 UTC m=+107.839921245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-s7zqk" (UID: "91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4") : secret "samples-operator-tls" not found
Sep 11 16:18:38.441768 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:38.441733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp" event={"ID":"88987724-bcf8-49cf-ad6c-45bc2eed9c0d","Type":"ContainerStarted","Data":"8609fb1d4f885dcad30a8d0b10bcaafbb4473ddbd518de449b670820fea4702c"}
Sep 11 16:18:38.470661 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:38.470604    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57bb67df64-nkcjp" podStartSLOduration=2.317945023 podStartE2EDuration="4.470588007s" totalImagesPullingTime="2.152642984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:38.468766389 +0000 UTC m=+100.962666201" watchObservedRunningTime="2026-09-11 16:18:38.470588007 +0000 UTC m=+100.964487819"
Sep 11 16:18:39.360772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.360674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:39.360943 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.360802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:39.360943 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:39.360909    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.361076 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:39.360962    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls podName:d5a56651-9a80-4d2a-9d3b-ab7d137682d8 nodeName:}" failed. No retries permitted until 2026-09-11 16:18:55.360948658 +0000 UTC m=+117.854848448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h6lpd" (UID: "d5a56651-9a80-4d2a-9d3b-ab7d137682d8") : secret "insights-runtime-extractor-tls" not found
Sep 11 16:18:39.361321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.361293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:39.979946 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.979906    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:39.980415 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.979996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:39.980484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:39.980416    2563 scope.go:117] "RemoveContainer" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"
Sep 11 16:18:39.980632 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:39.980607    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:18:40.348243 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:40.348160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tb8lh"
Sep 11 16:18:45.408137 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:45.408082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:45.410588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:45.410563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-s7zqk\" (UID: \"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:45.563772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:45.563720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"
Sep 11 16:18:45.835797 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:45.835762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk"]
Sep 11 16:18:46.469151 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:46.469109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk" event={"ID":"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4","Type":"ContainerStarted","Data":"dce77bee0ef8a7cca8a89be088c8fd8865bea1fd79bd78652345a89788136d60"}
Sep 11 16:18:47.475037 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:47.475009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk" event={"ID":"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4","Type":"ContainerStarted","Data":"f3fc14a94d0266e4f0ae66e1cd303d4fb291eb0229e6560898a85d7c14cff0ff"}
Sep 11 16:18:48.480172 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:48.480138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk" event={"ID":"91b5d633-e4e0-43a1-b8e6-cc63c4ab6ae4","Type":"ContainerStarted","Data":"9ba7223449fd16db853ba38d19b743f0d92b76ac0cdba94ae4e7f6293e50a3ee"}
Sep 11 16:18:48.502319 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:48.502272    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-s7zqk" podStartSLOduration=17.996067369 podStartE2EDuration="19.502256208s" totalImagesPullingTime="1.506188839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:48.500463036 +0000 UTC m=+110.994362848" watchObservedRunningTime="2026-09-11 16:18:48.502256208 +0000 UTC m=+110.996156055"
Sep 11 16:18:55.035979 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.035940    2563 scope.go:117] "RemoveContainer" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"
Sep 11 16:18:55.385233 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.385138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:55.387405 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.387381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5a56651-9a80-4d2a-9d3b-ab7d137682d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h6lpd\" (UID: \"d5a56651-9a80-4d2a-9d3b-ab7d137682d8\") " pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:55.503118 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503091    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:18:55.503501 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503488    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/1.log"
Sep 11 16:18:55.503554 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503519    2563 generic.go:356] "Generic (PLEG): container finished" podID="83bfe23b-dd43-457f-90f2-377a82c3d309" containerID="43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428" exitCode=255
Sep 11 16:18:55.503601 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-t629d" event={"ID":"83bfe23b-dd43-457f-90f2-377a82c3d309","Type":"ContainerDied","Data":"43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428"}
Sep 11 16:18:55.503642 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503632    2563 scope.go:117] "RemoveContainer" containerID="f285fe7a93eea93c5415cbb438f05e16c92171d42de8af6f92368590c708a242"
Sep 11 16:18:55.503950 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.503928    2563 scope.go:117] "RemoveContainer" containerID="43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428"
Sep 11 16:18:55.504153 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:55.504130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:18:55.659250 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.659167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h6lpd"
Sep 11 16:18:55.929602 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:55.929579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h6lpd"]
Sep 11 16:18:55.932053 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:55.932026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a56651_9a80_4d2a_9d3b_ab7d137682d8.slice/crio-f7b43d6dbc8c4cbfa379d27dc923881a91e85770fdc95f55006e814149d389eb WatchSource:0}: Error finding container f7b43d6dbc8c4cbfa379d27dc923881a91e85770fdc95f55006e814149d389eb: Status 404 returned error can't find the container with id f7b43d6dbc8c4cbfa379d27dc923881a91e85770fdc95f55006e814149d389eb
Sep 11 16:18:56.514457 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:56.514429    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:18:56.516222 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:56.516188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6lpd" event={"ID":"d5a56651-9a80-4d2a-9d3b-ab7d137682d8","Type":"ContainerStarted","Data":"b14e8d7fe7a00ae5b032999dbd65b66e2238f2d37d5d3b6ba7b2870c2670e82e"}
Sep 11 16:18:56.516336 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:56.516290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6lpd" event={"ID":"d5a56651-9a80-4d2a-9d3b-ab7d137682d8","Type":"ContainerStarted","Data":"f7b43d6dbc8c4cbfa379d27dc923881a91e85770fdc95f55006e814149d389eb"}
Sep 11 16:18:57.521397 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:57.521362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6lpd" event={"ID":"d5a56651-9a80-4d2a-9d3b-ab7d137682d8","Type":"ContainerStarted","Data":"9bf40fb193ca6741935f6174ee29f6485d34f138571bd9946286f0efe25fcd27"}
Sep 11 16:18:58.526837 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.526799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h6lpd" event={"ID":"d5a56651-9a80-4d2a-9d3b-ab7d137682d8","Type":"ContainerStarted","Data":"ebfe200372aa78f79cd5b592853e6367d98230c49efae6e337ccb1cfdc6d5fda"}
Sep 11 16:18:58.563470 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.563418    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h6lpd" podStartSLOduration=33.687282468 podStartE2EDuration="35.563402889s" totalImagesPullingTime="1.876120421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:18:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:58.562952801 +0000 UTC m=+121.056852612" watchObservedRunningTime="2026-09-11 16:18:58.563402889 +0000 UTC m=+121.057302701"
Sep 11 16:18:58.623493 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.623459    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-spghd"]
Sep 11 16:18:58.626765 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.626747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd"
Sep 11 16:18:58.630961 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.630941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j645f\""
Sep 11 16:18:58.638139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.638114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-spghd"]
Sep 11 16:18:58.711859 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.711823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpdf\" (UniqueName: \"kubernetes.io/projected/343bddec-7322-4515-9bf2-5b00e2e3a498-kube-api-access-wwpdf\") pod \"network-check-source-6f8b768455-spghd\" (UID: \"343bddec-7322-4515-9bf2-5b00e2e3a498\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd"
Sep 11 16:18:58.720786 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.720755    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"]
Sep 11 16:18:58.723734 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.723718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.728144 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.728120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 16:18:58.728274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.728169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 16:18:58.728338 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.728281    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ncg4w\""
Sep 11 16:18:58.734820 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.734798    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"]
Sep 11 16:18:58.737788 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.737769    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-556ff8d5bf-d2h28"]
Sep 11 16:18:58.737903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.737888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:18:58.740789 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.740768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.740924 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.740902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-klksm\""
Sep 11 16:18:58.741297 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.741280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 16:18:58.746530 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.746506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 16:18:58.746625 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.746598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nwnk6\""
Sep 11 16:18:58.746780 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.746758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 16:18:58.747305 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.746941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 16:18:58.748089 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.748042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"]
Sep 11 16:18:58.760485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.760466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 16:18:58.764701 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.764641    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"]
Sep 11 16:18:58.786465 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.786391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556ff8d5bf-d2h28"]
Sep 11 16:18:58.813111 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.813078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpdf\" (UniqueName: \"kubernetes.io/projected/343bddec-7322-4515-9bf2-5b00e2e3a498-kube-api-access-wwpdf\") pod \"network-check-source-6f8b768455-spghd\" (UID: \"343bddec-7322-4515-9bf2-5b00e2e3a498\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd"
Sep 11 16:18:58.813283 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.813117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd417906-9ea6-4cab-abd9-69af38a0132c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.813283 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.813159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a4d339c-7b66-41b2-838f-d8f6b9027896-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fmxkm\" (UID: \"6a4d339c-7b66-41b2-838f-d8f6b9027896\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:18:58.813283 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.813181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd417906-9ea6-4cab-abd9-69af38a0132c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.838114 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.838082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpdf\" (UniqueName: \"kubernetes.io/projected/343bddec-7322-4515-9bf2-5b00e2e3a498-kube-api-access-wwpdf\") pod \"network-check-source-6f8b768455-spghd\" (UID: \"343bddec-7322-4515-9bf2-5b00e2e3a498\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd"
Sep 11 16:18:58.914223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd417906-9ea6-4cab-abd9-69af38a0132c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.914404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-certificates\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-image-registry-private-configuration\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-installation-pull-secrets\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914404 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a4d339c-7b66-41b2-838f-d8f6b9027896-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fmxkm\" (UID: \"6a4d339c-7b66-41b2-838f-d8f6b9027896\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd417906-9ea6-4cab-abd9-69af38a0132c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-tls\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-ca-trust-extracted\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-trusted-ca\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr55l\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-kube-api-access-xr55l\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.914674 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.914650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-bound-sa-token\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:58.915048 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.915030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dd417906-9ea6-4cab-abd9-69af38a0132c-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.916542 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.916522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dd417906-9ea6-4cab-abd9-69af38a0132c-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6zvrz\" (UID: \"dd417906-9ea6-4cab-abd9-69af38a0132c\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:58.916890 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.916869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a4d339c-7b66-41b2-838f-d8f6b9027896-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-fmxkm\" (UID: \"6a4d339c-7b66-41b2-838f-d8f6b9027896\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:18:58.935605 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:58.935574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd"
Sep 11 16:18:59.015272 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr55l\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-kube-api-access-xr55l\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-bound-sa-token\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-certificates\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015329    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-image-registry-private-configuration\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-installation-pull-secrets\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-tls\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015716 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-ca-trust-extracted\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015716 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-trusted-ca\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.015895 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.015872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-ca-trust-extracted\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.016288 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.016262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-certificates\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.016370 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.016355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-trusted-ca\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.017864 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.017846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-registry-tls\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.018167 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.018145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-installation-pull-secrets\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.018267 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.018247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-image-registry-private-configuration\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.024991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.024969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-bound-sa-token\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.025081 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.025005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr55l\" (UniqueName: \"kubernetes.io/projected/8f17f094-b57a-4d2f-b8f8-69cb1d19f98c-kube-api-access-xr55l\") pod \"image-registry-556ff8d5bf-d2h28\" (UID: \"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c\") " pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.035984 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.035957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"
Sep 11 16:18:59.049249 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.049230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:18:59.054232 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.054212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.212266 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.212243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-spghd"]
Sep 11 16:18:59.214409 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:59.214381    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343bddec_7322_4515_9bf2_5b00e2e3a498.slice/crio-e00553b0991c226ed2d7f087e0ab6f485bb72a2a282457108cbab9ccff6266f3 WatchSource:0}: Error finding container e00553b0991c226ed2d7f087e0ab6f485bb72a2a282457108cbab9ccff6266f3: Status 404 returned error can't find the container with id e00553b0991c226ed2d7f087e0ab6f485bb72a2a282457108cbab9ccff6266f3
Sep 11 16:18:59.312002 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.311914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz"]
Sep 11 16:18:59.314783 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:59.314757    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd417906_9ea6_4cab_abd9_69af38a0132c.slice/crio-0cad609d6d05bbfd3568aa6f71a7e630889a324262e49f2f83e1af825158ba54 WatchSource:0}: Error finding container 0cad609d6d05bbfd3568aa6f71a7e630889a324262e49f2f83e1af825158ba54: Status 404 returned error can't find the container with id 0cad609d6d05bbfd3568aa6f71a7e630889a324262e49f2f83e1af825158ba54
Sep 11 16:18:59.347433 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.347408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"]
Sep 11 16:18:59.352229 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.352200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-556ff8d5bf-d2h28"]
Sep 11 16:18:59.352359 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:59.352263    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4d339c_7b66_41b2_838f_d8f6b9027896.slice/crio-1e0ebabb5adac11889d08e06459256f6759b401ddc5c70b1428305f387b9ab4e WatchSource:0}: Error finding container 1e0ebabb5adac11889d08e06459256f6759b401ddc5c70b1428305f387b9ab4e: Status 404 returned error can't find the container with id 1e0ebabb5adac11889d08e06459256f6759b401ddc5c70b1428305f387b9ab4e
Sep 11 16:18:59.355231 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:18:59.355210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f17f094_b57a_4d2f_b8f8_69cb1d19f98c.slice/crio-1c77c25141431a64bc4782a8029196ead6a1f417da8d66c0d6e45a560e37290c WatchSource:0}: Error finding container 1c77c25141431a64bc4782a8029196ead6a1f417da8d66c0d6e45a560e37290c: Status 404 returned error can't find the container with id 1c77c25141431a64bc4782a8029196ead6a1f417da8d66c0d6e45a560e37290c
Sep 11 16:18:59.531861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.531824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" event={"ID":"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c","Type":"ContainerStarted","Data":"4d8f400c4e267ae18dd25283052a144eb1e321572dc0ff1e1a9eebdefeecec46"}
Sep 11 16:18:59.532313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.531878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" event={"ID":"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c","Type":"ContainerStarted","Data":"1c77c25141431a64bc4782a8029196ead6a1f417da8d66c0d6e45a560e37290c"}
Sep 11 16:18:59.532313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.531930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:18:59.533545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.533520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz" event={"ID":"dd417906-9ea6-4cab-abd9-69af38a0132c","Type":"ContainerStarted","Data":"0cad609d6d05bbfd3568aa6f71a7e630889a324262e49f2f83e1af825158ba54"}
Sep 11 16:18:59.534721 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.534698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm" event={"ID":"6a4d339c-7b66-41b2-838f-d8f6b9027896","Type":"ContainerStarted","Data":"1e0ebabb5adac11889d08e06459256f6759b401ddc5c70b1428305f387b9ab4e"}
Sep 11 16:18:59.536294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.536274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd" event={"ID":"343bddec-7322-4515-9bf2-5b00e2e3a498","Type":"ContainerStarted","Data":"6c53db594b31ca8a9de2307178ba6294c20b8cfbe6bfdc2e42413469f5432435"}
Sep 11 16:18:59.536439 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.536306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd" event={"ID":"343bddec-7322-4515-9bf2-5b00e2e3a498","Type":"ContainerStarted","Data":"e00553b0991c226ed2d7f087e0ab6f485bb72a2a282457108cbab9ccff6266f3"}
Sep 11 16:18:59.570366 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.570250    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podStartSLOduration=1.5702280690000001 podStartE2EDuration="1.570228069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:59.56941259 +0000 UTC m=+122.063312401" watchObservedRunningTime="2026-09-11 16:18:59.570228069 +0000 UTC m=+122.064127886"
Sep 11 16:18:59.591732 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.591650    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-spghd" podStartSLOduration=1.591633901 podStartE2EDuration="1.591633901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:18:59.590388139 +0000 UTC m=+122.084287965" watchObservedRunningTime="2026-09-11 16:18:59.591633901 +0000 UTC m=+122.085533738"
Sep 11 16:18:59.979707 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.979641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:59.979893 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.979729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:18:59.980158 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:18:59.980141    2563 scope.go:117] "RemoveContainer" containerID="43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428"
Sep 11 16:18:59.980370 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:18:59.980343    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:19:01.545667 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.545622    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz" event={"ID":"dd417906-9ea6-4cab-abd9-69af38a0132c","Type":"ContainerStarted","Data":"9bf81dad2dcb64b7fa97c0e5590d8a8255f2e07f679e7b623502b6519efd252b"}
Sep 11 16:19:01.547173 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.547145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm" event={"ID":"6a4d339c-7b66-41b2-838f-d8f6b9027896","Type":"ContainerStarted","Data":"1573317946fedfb0f500c74c36ed7b0d70054ab108df8f8959cc6da563963495"}
Sep 11 16:19:01.547294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.547274    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:19:01.551873 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.551849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm"
Sep 11 16:19:01.564630 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.564583    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6zvrz" podStartSLOduration=2.13702776 podStartE2EDuration="3.564567559s" totalImagesPullingTime="1.427539799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:01.563393979 +0000 UTC m=+124.057293792" watchObservedRunningTime="2026-09-11 16:19:01.564567559 +0000 UTC m=+124.058467410"
Sep 11 16:19:01.580494 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:01.580445    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-fmxkm" podStartSLOduration=2.192206682 podStartE2EDuration="3.580428677s" totalImagesPullingTime="1.388221995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:01.579409229 +0000 UTC m=+124.073309040" watchObservedRunningTime="2026-09-11 16:19:01.580428677 +0000 UTC m=+124.074328488"
Sep 11 16:19:02.664260 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.664227    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"]
Sep 11 16:19:02.668731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.668714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.672462 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 16:19:02.672560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672470    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 16:19:02.672628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672558    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 16:19:02.672628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sf7sf\""
Sep 11 16:19:02.672819 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 16:19:02.672879 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.672806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 16:19:02.676995 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.676976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"]
Sep 11 16:19:02.848066 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.848010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.848066 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.848067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edba3433-9d2f-4a8d-9110-609e7aa82963-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.848301 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.848115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2fj\" (UniqueName: \"kubernetes.io/projected/edba3433-9d2f-4a8d-9110-609e7aa82963-kube-api-access-fj2fj\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.848301 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.848156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.948834 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.948806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2fj\" (UniqueName: \"kubernetes.io/projected/edba3433-9d2f-4a8d-9110-609e7aa82963-kube-api-access-fj2fj\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.948996 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.948841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.948996 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.948906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.948996 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.948928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edba3433-9d2f-4a8d-9110-609e7aa82963-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.949536 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.949515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edba3433-9d2f-4a8d-9110-609e7aa82963-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.951458 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.951429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.951553 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.951465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/edba3433-9d2f-4a8d-9110-609e7aa82963-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.958251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.958226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2fj\" (UniqueName: \"kubernetes.io/projected/edba3433-9d2f-4a8d-9110-609e7aa82963-kube-api-access-fj2fj\") pod \"prometheus-operator-68bcd7f987-9m2z4\" (UID: \"edba3433-9d2f-4a8d-9110-609e7aa82963\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:02.977738 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:02.977712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"
Sep 11 16:19:03.252588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:03.252509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4"]
Sep 11 16:19:03.255402 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:03.255376    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedba3433_9d2f_4a8d_9110_609e7aa82963.slice/crio-ec401b3b7a09f9c10866f6035b8b35f91d47a5b7794acac96e3f10fac3dac93c WatchSource:0}: Error finding container ec401b3b7a09f9c10866f6035b8b35f91d47a5b7794acac96e3f10fac3dac93c: Status 404 returned error can't find the container with id ec401b3b7a09f9c10866f6035b8b35f91d47a5b7794acac96e3f10fac3dac93c
Sep 11 16:19:03.555512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:03.555416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4" event={"ID":"edba3433-9d2f-4a8d-9110-609e7aa82963","Type":"ContainerStarted","Data":"ec401b3b7a09f9c10866f6035b8b35f91d47a5b7794acac96e3f10fac3dac93c"}
Sep 11 16:19:04.560416 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:04.560334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4" event={"ID":"edba3433-9d2f-4a8d-9110-609e7aa82963","Type":"ContainerStarted","Data":"377097db30d80f98793a7f6be8f7ce3e67a236a7e057c11824052f3dc9753291"}
Sep 11 16:19:04.560779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:04.560421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4" event={"ID":"edba3433-9d2f-4a8d-9110-609e7aa82963","Type":"ContainerStarted","Data":"18eb1ced2985caee3bbedd7e0b858e8da751d06eb39346bd5a507be287632088"}
Sep 11 16:19:04.583183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:04.583129    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-9m2z4" podStartSLOduration=1.532714702 podStartE2EDuration="2.583113352s" totalImagesPullingTime="1.05039865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:04.581193555 +0000 UTC m=+127.075093368" watchObservedRunningTime="2026-09-11 16:19:04.583113352 +0000 UTC m=+127.077013164"
Sep 11 16:19:06.118350 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.118319    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9km6q"]
Sep 11 16:19:06.123180 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.123161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.126316 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.126280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 16:19:06.127309 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.127284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 16:19:06.127559 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.127542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2bzxn\""
Sep 11 16:19:06.127636 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.127564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 16:19:06.274068 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-metrics-client-ca\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274252 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-wtmp\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274252 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28dkc\" (UniqueName: \"kubernetes.io/projected/cf4ea204-cf62-4021-b549-c5dc14c5e429-kube-api-access-28dkc\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274252 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-sys\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274252 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-textfile\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-root\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-tls\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.274451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.274384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-accelerators-collector-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-wtmp\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28dkc\" (UniqueName: \"kubernetes.io/projected/cf4ea204-cf62-4021-b549-c5dc14c5e429-kube-api-access-28dkc\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-sys\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-textfile\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-sys\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-wtmp\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-root\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf4ea204-cf62-4021-b549-c5dc14c5e429-root\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-tls\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.375991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-accelerators-collector-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.376313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.375995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-textfile\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.376313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.376004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-metrics-client-ca\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.376417 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.376391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-accelerators-collector-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.376515 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.376489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf4ea204-cf62-4021-b549-c5dc14c5e429-metrics-client-ca\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.378103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.378081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-tls\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.378202 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.378188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf4ea204-cf62-4021-b549-c5dc14c5e429-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.384632 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.384612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28dkc\" (UniqueName: \"kubernetes.io/projected/cf4ea204-cf62-4021-b549-c5dc14c5e429-kube-api-access-28dkc\") pod \"node-exporter-9km6q\" (UID: \"cf4ea204-cf62-4021-b549-c5dc14c5e429\") " pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.431802 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.431766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9km6q"
Sep 11 16:19:06.443191 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:06.443165    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4ea204_cf62_4021_b549_c5dc14c5e429.slice/crio-b42019edac855de6335711fa6da621187d06bf8b71ffc9db46e7dba8182e457b WatchSource:0}: Error finding container b42019edac855de6335711fa6da621187d06bf8b71ffc9db46e7dba8182e457b: Status 404 returned error can't find the container with id b42019edac855de6335711fa6da621187d06bf8b71ffc9db46e7dba8182e457b
Sep 11 16:19:06.567506 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:06.567473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9km6q" event={"ID":"cf4ea204-cf62-4021-b549-c5dc14c5e429","Type":"ContainerStarted","Data":"b42019edac855de6335711fa6da621187d06bf8b71ffc9db46e7dba8182e457b"}
Sep 11 16:19:07.789125 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:07.789088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:19:07.791325 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:07.791299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f-metrics-certs\") pod \"network-metrics-daemon-zjf55\" (UID: \"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f\") " pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:19:07.948754 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:07.948723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks2m\""
Sep 11 16:19:07.956262 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:07.956237    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zjf55"
Sep 11 16:19:08.047870 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.047842    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"]
Sep 11 16:19:08.068789 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.068757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"]
Sep 11 16:19:08.068935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.068866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.072017 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.071994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 16:19:08.072017 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.072011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pxlxn\""
Sep 11 16:19:08.072293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.071995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 16:19:08.072293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.072068    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 16:19:08.072293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.071993    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 16:19:08.072293 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.072100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 16:19:08.072809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.072776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-59tsqlb37r51n\""
Sep 11 16:19:08.191465 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191465 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-grpc-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5r89\" (UniqueName: \"kubernetes.io/projected/58756fe8-9bbb-47cd-a724-76449798f471-kube-api-access-v5r89\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.191869 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.191750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58756fe8-9bbb-47cd-a724-76449798f471-metrics-client-ca\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.238126 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.238073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zjf55"]
Sep 11 16:19:08.240713 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:08.240665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d0cf25_71fd_4b8d_ab14_7c6f34f1d22f.slice/crio-132006c845fe9c652e2df2009e911197f347bacb59dfdefc0e879a71191f3338 WatchSource:0}: Error finding container 132006c845fe9c652e2df2009e911197f347bacb59dfdefc0e879a71191f3338: Status 404 returned error can't find the container with id 132006c845fe9c652e2df2009e911197f347bacb59dfdefc0e879a71191f3338
Sep 11 16:19:08.292184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58756fe8-9bbb-47cd-a724-76449798f471-metrics-client-ca\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-grpc-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.292489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.292308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5r89\" (UniqueName: \"kubernetes.io/projected/58756fe8-9bbb-47cd-a724-76449798f471-kube-api-access-v5r89\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.293052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.293021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58756fe8-9bbb-47cd-a724-76449798f471-metrics-client-ca\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.294838 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.294808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.294919 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.294884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.295217 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.295198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.295251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.295225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.295251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.295234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-thanos-querier-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.295313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.295292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58756fe8-9bbb-47cd-a724-76449798f471-secret-grpc-tls\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.302421 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.302403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5r89\" (UniqueName: \"kubernetes.io/projected/58756fe8-9bbb-47cd-a724-76449798f471-kube-api-access-v5r89\") pod \"thanos-querier-86bb9dc985-qgvpg\" (UID: \"58756fe8-9bbb-47cd-a724-76449798f471\") " pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.414419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.414379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:08.576791 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.576672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjf55" event={"ID":"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f","Type":"ContainerStarted","Data":"132006c845fe9c652e2df2009e911197f347bacb59dfdefc0e879a71191f3338"}
Sep 11 16:19:08.578215 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.578191    2563 generic.go:356] "Generic (PLEG): container finished" podID="cf4ea204-cf62-4021-b549-c5dc14c5e429" containerID="2c405f93b72ddf83b66b4e31a9191dbc0024b300bfb49d03203babaf67b78d63" exitCode=0
Sep 11 16:19:08.578317 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.578250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9km6q" event={"ID":"cf4ea204-cf62-4021-b549-c5dc14c5e429","Type":"ContainerDied","Data":"2c405f93b72ddf83b66b4e31a9191dbc0024b300bfb49d03203babaf67b78d63"}
Sep 11 16:19:08.717862 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:08.717826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"]
Sep 11 16:19:08.721202 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:08.721172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58756fe8_9bbb_47cd_a724_76449798f471.slice/crio-3f79f36c76e250023ce4ec3d812539919fc29d04586c18f607958eb5328c2444 WatchSource:0}: Error finding container 3f79f36c76e250023ce4ec3d812539919fc29d04586c18f607958eb5328c2444: Status 404 returned error can't find the container with id 3f79f36c76e250023ce4ec3d812539919fc29d04586c18f607958eb5328c2444
Sep 11 16:19:09.584893 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:09.584843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9km6q" event={"ID":"cf4ea204-cf62-4021-b549-c5dc14c5e429","Type":"ContainerStarted","Data":"6583151d43061e784e92c22e805b9bb8a4d715cd3bc3cb7c3e41a3e10cd19073"}
Sep 11 16:19:09.585376 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:09.584917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9km6q" event={"ID":"cf4ea204-cf62-4021-b549-c5dc14c5e429","Type":"ContainerStarted","Data":"d5e14581fab7399ca663eac69cca737fc37acae6dc3543d92cadd334c8060e95"}
Sep 11 16:19:09.586211 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:09.586184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"3f79f36c76e250023ce4ec3d812539919fc29d04586c18f607958eb5328c2444"}
Sep 11 16:19:09.608997 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:09.608955    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9km6q" podStartSLOduration=2.131896422 podStartE2EDuration="3.608937923s" totalImagesPullingTime="1.402232855s" totalInitContainerRuntime="74.808646ms" isStatefulPod=true podCreationTimestamp="2026-09-11 16:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:09.607609078 +0000 UTC m=+132.101508892" watchObservedRunningTime="2026-09-11 16:19:09.608937923 +0000 UTC m=+132.102837734"
Sep 11 16:19:10.594296 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.594254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjf55" event={"ID":"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f","Type":"ContainerStarted","Data":"6e7b316253ac5b69a4548d39ed31d3595dd19e75ba4c7d49137a850071517e7c"}
Sep 11 16:19:10.594760 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.594321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zjf55" event={"ID":"13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f","Type":"ContainerStarted","Data":"756b0ec58acaae5a1df4c702cde2dc9db8d756f7715ee441a6e9ac83c4de6468"}
Sep 11 16:19:10.618918 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.615710    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zjf55" podStartSLOduration=131.259655989 podStartE2EDuration="2m12.615666749s" totalImagesPullingTime="1.35601076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:10.614612188 +0000 UTC m=+133.108512002" watchObservedRunningTime="2026-09-11 16:19:10.615666749 +0000 UTC m=+133.109566556"
Sep 11 16:19:10.816849 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.816810    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"]
Sep 11 16:19:10.820049 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.820022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:10.823006 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.822979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 16:19:10.823136 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.823040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-768bw\""
Sep 11 16:19:10.828931 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.828899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"]
Sep 11 16:19:10.914696 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.914604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f573c87-ea54-4c21-a781-79a4f1933a0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:10.914696 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:10.914669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cwt\" (UniqueName: \"kubernetes.io/projected/1f573c87-ea54-4c21-a781-79a4f1933a0c-kube-api-access-x2cwt\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.015382 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.015353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cwt\" (UniqueName: \"kubernetes.io/projected/1f573c87-ea54-4c21-a781-79a4f1933a0c-kube-api-access-x2cwt\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.015509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.015408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f573c87-ea54-4c21-a781-79a4f1933a0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.017646 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.017618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1f573c87-ea54-4c21-a781-79a4f1933a0c-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.029074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.029052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cwt\" (UniqueName: \"kubernetes.io/projected/1f573c87-ea54-4c21-a781-79a4f1933a0c-kube-api-access-x2cwt\") pod \"monitoring-plugin-5d85778b9d-q5sm9\" (UID: \"1f573c87-ea54-4c21-a781-79a4f1933a0c\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.133280 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.133252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:11.409101 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.409067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"]
Sep 11 16:19:11.411841 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:11.411816    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f573c87_ea54_4c21_a781_79a4f1933a0c.slice/crio-0b47a8757e5ff00138a6e106342584c813be88bac84785d804402c3beef9740f WatchSource:0}: Error finding container 0b47a8757e5ff00138a6e106342584c813be88bac84785d804402c3beef9740f: Status 404 returned error can't find the container with id 0b47a8757e5ff00138a6e106342584c813be88bac84785d804402c3beef9740f
Sep 11 16:19:11.600873 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.600833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"7f90918391f4483a85eeb7663f468acbc5c96d7e77a5b7c73e948e8e1899e973"}
Sep 11 16:19:11.601335 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.600915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"9b21dd322e1230277d7e1c2439d38f799d8f6bfa44166b21a038491efd86c0c7"}
Sep 11 16:19:11.601335 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.600955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"c69f06b8686ffce381b0710f9e8672864221e0435941164b366120b1ff942c21"}
Sep 11 16:19:11.602228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:11.602187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9" event={"ID":"1f573c87-ea54-4c21-a781-79a4f1933a0c","Type":"ContainerStarted","Data":"0b47a8757e5ff00138a6e106342584c813be88bac84785d804402c3beef9740f"}
Sep 11 16:19:12.613442 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:12.613404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"248cd047590aab30a6d7db85e83ab6adaa03d0af6dcd8b7b3fb6be4e22ec190b"}
Sep 11 16:19:12.613903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:12.613487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"193b927c22c2b2699fc525a1d281bb1b40ca73d00e5a43d1c569810b697486d2"}
Sep 11 16:19:12.613903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:12.613527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" event={"ID":"58756fe8-9bbb-47cd-a724-76449798f471","Type":"ContainerStarted","Data":"4f9fd2dd18eaaa97e24f543164ae963c9287bce9830cf34dd69c66bc202e4c44"}
Sep 11 16:19:12.613903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:12.613631    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:12.639223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:12.639167    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg" podStartSLOduration=1.774342136 podStartE2EDuration="4.639150571s" totalImagesPullingTime="2.864808435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:12.636472526 +0000 UTC m=+135.130372360" watchObservedRunningTime="2026-09-11 16:19:12.639150571 +0000 UTC m=+135.133050382"
Sep 11 16:19:13.037050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:13.037030    2563 scope.go:117] "RemoveContainer" containerID="43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428"
Sep 11 16:19:13.037257 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:19:13.037226    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-t629d_openshift-console-operator(83bfe23b-dd43-457f-90f2-377a82c3d309)\"" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podUID="83bfe23b-dd43-457f-90f2-377a82c3d309"
Sep 11 16:19:13.618941 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:13.618903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9" event={"ID":"1f573c87-ea54-4c21-a781-79a4f1933a0c","Type":"ContainerStarted","Data":"e7ba47c78b4b16509dc9bb6a28973604798b9d1b7629974c79b9fc490ec24d15"}
Sep 11 16:19:13.619379 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:13.619196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:13.623867 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:13.623845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9"
Sep 11 16:19:13.637785 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:13.637743    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-q5sm9" podStartSLOduration=2.072529776 podStartE2EDuration="3.637702201s" totalImagesPullingTime="1.565172425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:13.637072243 +0000 UTC m=+136.130972056" watchObservedRunningTime="2026-09-11 16:19:13.637702201 +0000 UTC m=+136.131602013"
Sep 11 16:19:18.624612 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:18.624583    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86bb9dc985-qgvpg"
Sep 11 16:19:19.058695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:19.058648    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:19.058858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:19.058726    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:20.545395 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:20.545357    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:20.545780 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:20.545412    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:28.040899 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.040872    2563 scope.go:117] "RemoveContainer" containerID="43dcfea67d2fdb81fd37e41a18c9e98017bd3c9d2659b0f28aba9748f7af4428"
Sep 11 16:19:28.677483 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.677456    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:19:28.677662 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.677511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-t629d" event={"ID":"83bfe23b-dd43-457f-90f2-377a82c3d309","Type":"ContainerStarted","Data":"cf01afc7dd77a94974a050857a6d308d040a74214e6023a22458d2ee6db325bb"}
Sep 11 16:19:28.677818 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.677794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:19:28.682371 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.682343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-t629d"
Sep 11 16:19:28.699080 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:28.699034    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-t629d" podStartSLOduration=56.971433238 podStartE2EDuration="59.699020312s" totalImagesPullingTime="2.727587074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:28.697406868 +0000 UTC m=+151.191306680" watchObservedRunningTime="2026-09-11 16:19:28.699020312 +0000 UTC m=+151.192920123"
Sep 11 16:19:29.058526 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:29.058492    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:29.058933 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:29.058548    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:30.545290 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:30.545252    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:30.545702 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:30.545311    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:35.883862 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:19:35.883815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6hj5n" podUID="38a822e7-2ba9-4d1e-b494-1056a88c7af8"
Sep 11 16:19:36.705823 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:36.705792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:39.057927 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.057894    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:39.058343 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.057950    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:39.058343 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.057984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:19:39.058459 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.058390    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"4d8f400c4e267ae18dd25283052a144eb1e321572dc0ff1e1a9eebdefeecec46"} pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 11 16:19:39.062324 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.062296    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:39.062456 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.062346    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:39.248815 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.248776    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:19:39.257079 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.257056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.260927 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.260880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 16:19:39.261060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.260955    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 16:19:39.261060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.260963    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 16:19:39.261060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.260963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 16:19:39.261060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.261024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 16:19:39.261060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.261037    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 16:19:39.261297 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.261220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 16:19:39.261297 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.261254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q8hlt\""
Sep 11 16:19:39.264555 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.264537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:19:39.326000 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.325906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.326000 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.325951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghvv\" (UniqueName: \"kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.326218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.326024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.326218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.326075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.326218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.326101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.326218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.326173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426635 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghvv\" (UniqueName: \"kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.426826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.426793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.427353 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.427329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.427445 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.427421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.427504 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.427449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.429168 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.429148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.429276 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.429258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.435517 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.435486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghvv\" (UniqueName: \"kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv\") pod \"console-544486f6bc-64pw9\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") " pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.566333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.566298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:39.839307 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:39.839231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:19:39.842304 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:39.842272    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f87cf4_b9fd_4497_88c1_f3b5339b8739.slice/crio-21ebdc0d9dac871abc55d9236e67303754411164ba4e5c17c110b1e74de634a4 WatchSource:0}: Error finding container 21ebdc0d9dac871abc55d9236e67303754411164ba4e5c17c110b1e74de634a4: Status 404 returned error can't find the container with id 21ebdc0d9dac871abc55d9236e67303754411164ba4e5c17c110b1e74de634a4
Sep 11 16:19:40.722797 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:40.722757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544486f6bc-64pw9" event={"ID":"f1f87cf4-b9fd-4497-88c1-f3b5339b8739","Type":"ContainerStarted","Data":"21ebdc0d9dac871abc55d9236e67303754411164ba4e5c17c110b1e74de634a4"}
Sep 11 16:19:40.838512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:40.838469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:40.841196 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:40.841170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/38a822e7-2ba9-4d1e-b494-1056a88c7af8-metrics-tls\") pod \"dns-default-6hj5n\" (UID: \"38a822e7-2ba9-4d1e-b494-1056a88c7af8\") " pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:40.909063 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:40.909034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-244pg\""
Sep 11 16:19:40.916540 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:40.916503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:41.201322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:41.201292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hj5n"]
Sep 11 16:19:41.203776 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:41.203732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a822e7_2ba9_4d1e_b494_1056a88c7af8.slice/crio-6309539c96cca716baeb9e9b34909e9c3753c759b4a16430b396f9b3f6ee3d59 WatchSource:0}: Error finding container 6309539c96cca716baeb9e9b34909e9c3753c759b4a16430b396f9b3f6ee3d59: Status 404 returned error can't find the container with id 6309539c96cca716baeb9e9b34909e9c3753c759b4a16430b396f9b3f6ee3d59
Sep 11 16:19:41.729825 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:41.729781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hj5n" event={"ID":"38a822e7-2ba9-4d1e-b494-1056a88c7af8","Type":"ContainerStarted","Data":"6309539c96cca716baeb9e9b34909e9c3753c759b4a16430b396f9b3f6ee3d59"}
Sep 11 16:19:43.742591 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:43.742453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544486f6bc-64pw9" event={"ID":"f1f87cf4-b9fd-4497-88c1-f3b5339b8739","Type":"ContainerStarted","Data":"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"}
Sep 11 16:19:43.745292 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:43.745158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hj5n" event={"ID":"38a822e7-2ba9-4d1e-b494-1056a88c7af8","Type":"ContainerStarted","Data":"8bb7fca7abd3fac081ff44396b8c5fcc2a95e415dfcec2995c6b8d0fb17374ee"}
Sep 11 16:19:43.762726 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:43.762456    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-544486f6bc-64pw9" podStartSLOduration=0.997518711 podStartE2EDuration="4.762433903s" totalImagesPullingTime="3.764915192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:43.760754029 +0000 UTC m=+166.254653842" watchObservedRunningTime="2026-09-11 16:19:43.762433903 +0000 UTC m=+166.256333716"
Sep 11 16:19:44.750673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:44.750628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hj5n" event={"ID":"38a822e7-2ba9-4d1e-b494-1056a88c7af8","Type":"ContainerStarted","Data":"d3b7e5e460d5bcd122670e9aecf532b169433a5660fc5ed3c1020fa53f1775fe"}
Sep 11 16:19:44.771174 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:44.771121    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6hj5n" podStartSLOduration=130.367474449 podStartE2EDuration="2m12.771105292s" totalImagesPullingTime="2.403630843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:44.770093399 +0000 UTC m=+167.263993213" watchObservedRunningTime="2026-09-11 16:19:44.771105292 +0000 UTC m=+167.265005103"
Sep 11 16:19:45.754821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:45.754792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:48.419559 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.419523    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:19:48.422859 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.422841    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.430807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.430782    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 16:19:48.436034 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.436008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:19:48.501391 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501391 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5smw\" (UniqueName: \"kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.501594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.501585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.602840 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.602788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.602840 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.602848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603086 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.602884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603086 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.602981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603184 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5smw\" (UniqueName: \"kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603734 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603860 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.603860 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.603828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.604097 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.604068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.605464 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.605434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.605611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.605592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.611756 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.611733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5smw\" (UniqueName: \"kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw\") pod \"console-55764b64cf-fmgwb\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") " pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:48.732526 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:48.732490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:49.018446 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.018421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:19:49.020712 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:19:49.020668    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66a369d_0b5e_46ed_87bf_725ea9ccae54.slice/crio-9900605eb278d862625e10652687f7268618c8c00f95a2dc1681c9fe78da636a WatchSource:0}: Error finding container 9900605eb278d862625e10652687f7268618c8c00f95a2dc1681c9fe78da636a: Status 404 returned error can't find the container with id 9900605eb278d862625e10652687f7268618c8c00f95a2dc1681c9fe78da636a
Sep 11 16:19:49.062411 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.062378    2563 patch_prober.go:28] interesting pod/image-registry-556ff8d5bf-d2h28 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 16:19:49.062535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.062439    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:19:49.566784 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.566748    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:49.567219 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.566808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:49.571809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.571782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:49.772455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.772419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55764b64cf-fmgwb" event={"ID":"f66a369d-0b5e-46ed-87bf-725ea9ccae54","Type":"ContainerStarted","Data":"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"}
Sep 11 16:19:49.772719 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.772475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55764b64cf-fmgwb" event={"ID":"f66a369d-0b5e-46ed-87bf-725ea9ccae54","Type":"ContainerStarted","Data":"9900605eb278d862625e10652687f7268618c8c00f95a2dc1681c9fe78da636a"}
Sep 11 16:19:49.776486 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.776460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:19:49.791454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:49.791389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55764b64cf-fmgwb" podStartSLOduration=1.791371743 podStartE2EDuration="1.791371743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:49.790988775 +0000 UTC m=+172.284888589" watchObservedRunningTime="2026-09-11 16:19:49.791371743 +0000 UTC m=+172.285271556"
Sep 11 16:19:52.787303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:52.787266    2563 generic.go:356] "Generic (PLEG): container finished" podID="4de17a84-cf83-4cee-a71b-d70c7cfba916" containerID="45cff109c84307326c138ca2364fcba53f8f71866b886121357917568270c854" exitCode=0
Sep 11 16:19:52.787671 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:52.787330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8drrk" event={"ID":"4de17a84-cf83-4cee-a71b-d70c7cfba916","Type":"ContainerDied","Data":"45cff109c84307326c138ca2364fcba53f8f71866b886121357917568270c854"}
Sep 11 16:19:52.787671 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:52.787657    2563 scope.go:117] "RemoveContainer" containerID="45cff109c84307326c138ca2364fcba53f8f71866b886121357917568270c854"
Sep 11 16:19:53.793433 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:53.793402    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9678d83-ed8e-41f3-9562-1f562984b9a2" containerID="332bfa05688384cb41e61e3b562afe4122f40bebc7aa19580b8aceaeaaf3ff14" exitCode=0
Sep 11 16:19:53.793883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:53.793474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n" event={"ID":"f9678d83-ed8e-41f3-9562-1f562984b9a2","Type":"ContainerDied","Data":"332bfa05688384cb41e61e3b562afe4122f40bebc7aa19580b8aceaeaaf3ff14"}
Sep 11 16:19:53.793883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:53.793853    2563 scope.go:117] "RemoveContainer" containerID="332bfa05688384cb41e61e3b562afe4122f40bebc7aa19580b8aceaeaaf3ff14"
Sep 11 16:19:53.795649 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:53.795627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-8drrk" event={"ID":"4de17a84-cf83-4cee-a71b-d70c7cfba916","Type":"ContainerStarted","Data":"3e249ed8f49e7e2294b119d36886c383acb2af27ce271a9f7be6f2b1cf6f2ab7"}
Sep 11 16:19:54.801727 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:54.801691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-6fq7n" event={"ID":"f9678d83-ed8e-41f3-9562-1f562984b9a2","Type":"ContainerStarted","Data":"da68cdad2b351d607b03385ddc661158885aa67202966d032e15d7f52eb21cfb"}
Sep 11 16:19:55.761086 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:55.761054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6hj5n"
Sep 11 16:19:58.733491 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:58.733441    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:58.734240 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:58.733528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:58.739927 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:58.739899    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:58.823812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:58.823779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:19:58.888033 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:58.888002    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:19:59.063012 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:19:59.062946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:20:03.840104 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:03.840065    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a" containerID="ed9727316974231ee999f603d13ab7a89381835acf65daec02bbf25efafb59af" exitCode=0
Sep 11 16:20:03.840467 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:03.840146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5" event={"ID":"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a","Type":"ContainerDied","Data":"ed9727316974231ee999f603d13ab7a89381835acf65daec02bbf25efafb59af"}
Sep 11 16:20:03.840504 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:03.840470    2563 scope.go:117] "RemoveContainer" containerID="ed9727316974231ee999f603d13ab7a89381835acf65daec02bbf25efafb59af"
Sep 11 16:20:04.077468 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:04.077419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" podUID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerName="registry" containerID="cri-o://4d8f400c4e267ae18dd25283052a144eb1e321572dc0ff1e1a9eebdefeecec46" gracePeriod=30
Sep 11 16:20:04.845689 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:04.845637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l6qv5" event={"ID":"0b30d5ea-db78-4b29-b0a3-b5c4954ffd2a","Type":"ContainerStarted","Data":"a0761b4d04e064a1dd9c019549723077fc0c221b7a1698fff1a5e6aa4b1d1522"}
Sep 11 16:20:05.851384 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:05.851350    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f17f094-b57a-4d2f-b8f8-69cb1d19f98c" containerID="4d8f400c4e267ae18dd25283052a144eb1e321572dc0ff1e1a9eebdefeecec46" exitCode=0
Sep 11 16:20:05.851775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:05.851440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" event={"ID":"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c","Type":"ContainerDied","Data":"4d8f400c4e267ae18dd25283052a144eb1e321572dc0ff1e1a9eebdefeecec46"}
Sep 11 16:20:05.851775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:05.851490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28" event={"ID":"8f17f094-b57a-4d2f-b8f8-69cb1d19f98c","Type":"ContainerStarted","Data":"40e3c305acd9256cb95c994a72469edb70eee02d68fb9f6cf776b734a333e963"}
Sep 11 16:20:05.851775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:05.851511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:20:23.908316 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:23.908267    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-544486f6bc-64pw9" podUID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" containerName="console" containerID="cri-o://f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6" gracePeriod=15
Sep 11 16:20:24.155330 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.155303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:20:24.311516 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311482    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.311728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.311728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hghvv\" (UniqueName: \"kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.311728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.311728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311616    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.311728 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.311663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert\") pod \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\" (UID: \"f1f87cf4-b9fd-4497-88c1-f3b5339b8739\") "
Sep 11 16:20:24.312119 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.312091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:24.312195 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.312114    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:24.312195 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.312100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config" (OuterVolumeSpecName: "console-config") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:20:24.313916 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.313881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:24.314011 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.313937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv" (OuterVolumeSpecName: "kube-api-access-hghvv") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "kube-api-access-hghvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:20:24.314011 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.313994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1f87cf4-b9fd-4497-88c1-f3b5339b8739" (UID: "f1f87cf4-b9fd-4497-88c1-f3b5339b8739"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:20:24.412717 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412645    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hghvv\" (UniqueName: \"kubernetes.io/projected/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-kube-api-access-hghvv\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.412717 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412713    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-service-ca\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.412717 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412723    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-oauth-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.412717 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412731    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.412985 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412739    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-oauth-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.412985 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.412748    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1f87cf4-b9fd-4497-88c1-f3b5339b8739-console-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:20:24.926517 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.926477    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" containerID="f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6" exitCode=0
Sep 11 16:20:24.926967 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.926526    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-544486f6bc-64pw9"
Sep 11 16:20:24.926967 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.926550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544486f6bc-64pw9" event={"ID":"f1f87cf4-b9fd-4497-88c1-f3b5339b8739","Type":"ContainerDied","Data":"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"}
Sep 11 16:20:24.926967 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.926600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-544486f6bc-64pw9" event={"ID":"f1f87cf4-b9fd-4497-88c1-f3b5339b8739","Type":"ContainerDied","Data":"21ebdc0d9dac871abc55d9236e67303754411164ba4e5c17c110b1e74de634a4"}
Sep 11 16:20:24.926967 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.926627    2563 scope.go:117] "RemoveContainer" containerID="f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"
Sep 11 16:20:24.935297 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.935275    2563 scope.go:117] "RemoveContainer" containerID="f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"
Sep 11 16:20:24.935572 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:20:24.935553    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6\": container with ID starting with f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6 not found: ID does not exist" containerID="f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"
Sep 11 16:20:24.935637 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.935578    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6"} err="failed to get container status \"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6\": rpc error: code = NotFound desc = could not find container \"f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6\": container with ID starting with f0bf983b07171ef07284b5c0e6bad3cfc3002bb9376a00f8b88f68be42ce89e6 not found: ID does not exist"
Sep 11 16:20:24.948326 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.948300    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:20:24.952599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:24.952579    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-544486f6bc-64pw9"]
Sep 11 16:20:26.041213 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:26.041182    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" path="/var/lib/kubelet/pods/f1f87cf4-b9fd-4497-88c1-f3b5339b8739/volumes"
Sep 11 16:20:26.860003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:26.859976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-556ff8d5bf-d2h28"
Sep 11 16:20:28.739222 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.739191    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:20:28.739591 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.739572    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" containerName="console"
Sep 11 16:20:28.739636 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.739594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" containerName="console"
Sep 11 16:20:28.739699 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.739668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f87cf4-b9fd-4497-88c1-f3b5339b8739" containerName="console"
Sep 11 16:20:28.745965 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.745941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.762903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.762876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:20:28.845884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.845849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.845889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.845910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.845961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw8f\" (UniqueName: \"kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.845993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.846016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.846060 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.846038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946442 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946442 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946710 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946710 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946710 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946710 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.946710 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.946587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw8f\" (UniqueName: \"kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.947366 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.947334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.947485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.947335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.947485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.947425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.947627 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.947609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.948936 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.948918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.949012 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.948975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:28.955213 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:28.955195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw8f\" (UniqueName: \"kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f\") pod \"console-764c4dc4f7-272zd\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") " pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:29.055036 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:29.054957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:29.328112 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:29.328086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:20:29.330364 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:20:29.330326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a23301a_4605_4f5d_b045_ee7c0a1b3b25.slice/crio-66d555c96aa0fc7bb51ecebd777085e196073a49c3c2a6f7633aa9749c4bda33 WatchSource:0}: Error finding container 66d555c96aa0fc7bb51ecebd777085e196073a49c3c2a6f7633aa9749c4bda33: Status 404 returned error can't find the container with id 66d555c96aa0fc7bb51ecebd777085e196073a49c3c2a6f7633aa9749c4bda33
Sep 11 16:20:29.951389 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:29.951355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764c4dc4f7-272zd" event={"ID":"4a23301a-4605-4f5d-b045-ee7c0a1b3b25","Type":"ContainerStarted","Data":"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"}
Sep 11 16:20:29.951798 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:29.951403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764c4dc4f7-272zd" event={"ID":"4a23301a-4605-4f5d-b045-ee7c0a1b3b25","Type":"ContainerStarted","Data":"66d555c96aa0fc7bb51ecebd777085e196073a49c3c2a6f7633aa9749c4bda33"}
Sep 11 16:20:29.970764 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:29.970712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-764c4dc4f7-272zd" podStartSLOduration=1.970695381 podStartE2EDuration="1.970695381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:29.969548532 +0000 UTC m=+212.463448344" watchObservedRunningTime="2026-09-11 16:20:29.970695381 +0000 UTC m=+212.464595184"
Sep 11 16:20:39.055092 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:39.055061    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:39.055581 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:39.055152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:39.061110 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:39.061085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:39.995329 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:39.995297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:20:40.043076 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:20:40.043046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:21:05.064948 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.064892    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55764b64cf-fmgwb" podUID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" containerName="console" containerID="cri-o://1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49" gracePeriod=15
Sep 11 16:21:05.310374 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.310347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:21:05.426096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5smw\" (UniqueName: \"kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426179    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426242    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426276    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle\") pod \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\" (UID: \"f66a369d-0b5e-46ed-87bf-725ea9ccae54\") "
Sep 11 16:21:05.426519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca" (OuterVolumeSpecName: "service-ca") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:05.426611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:05.426712 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426669    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-oauth-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.426786 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426717    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-service-ca\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.426786 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config" (OuterVolumeSpecName: "console-config") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:05.426873 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.426780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:05.428435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.428413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:05.428756 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.428732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw" (OuterVolumeSpecName: "kube-api-access-w5smw") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "kube-api-access-w5smw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:05.428853 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.428812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f66a369d-0b5e-46ed-87bf-725ea9ccae54" (UID: "f66a369d-0b5e-46ed-87bf-725ea9ccae54"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:05.527261 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.527218    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5smw\" (UniqueName: \"kubernetes.io/projected/f66a369d-0b5e-46ed-87bf-725ea9ccae54-kube-api-access-w5smw\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.527261 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.527252    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-oauth-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.527261 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.527262    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.527261 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.527272    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f66a369d-0b5e-46ed-87bf-725ea9ccae54-console-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:05.527509 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:05.527281    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f66a369d-0b5e-46ed-87bf-725ea9ccae54-trusted-ca-bundle\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:21:06.093265 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.093230    2563 generic.go:356] "Generic (PLEG): container finished" podID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" containerID="1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49" exitCode=0
Sep 11 16:21:06.093673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.093276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55764b64cf-fmgwb"
Sep 11 16:21:06.093673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.093298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55764b64cf-fmgwb" event={"ID":"f66a369d-0b5e-46ed-87bf-725ea9ccae54","Type":"ContainerDied","Data":"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"}
Sep 11 16:21:06.093673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.093335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55764b64cf-fmgwb" event={"ID":"f66a369d-0b5e-46ed-87bf-725ea9ccae54","Type":"ContainerDied","Data":"9900605eb278d862625e10652687f7268618c8c00f95a2dc1681c9fe78da636a"}
Sep 11 16:21:06.093673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.093353    2563 scope.go:117] "RemoveContainer" containerID="1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"
Sep 11 16:21:06.101375 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.101356    2563 scope.go:117] "RemoveContainer" containerID="1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"
Sep 11 16:21:06.101649 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:21:06.101632    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49\": container with ID starting with 1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49 not found: ID does not exist" containerID="1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"
Sep 11 16:21:06.101708 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.101657    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49"} err="failed to get container status \"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49\": rpc error: code = NotFound desc = could not find container \"1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49\": container with ID starting with 1e0da383a8af979e8f5731ba1d948b7cd95bbe12e8678074564ae3c41ea0be49 not found: ID does not exist"
Sep 11 16:21:06.110287 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.110266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:21:06.122147 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:06.122111    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55764b64cf-fmgwb"]
Sep 11 16:21:08.040476 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:08.040440    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" path="/var/lib/kubelet/pods/f66a369d-0b5e-46ed-87bf-725ea9ccae54/volumes"
Sep 11 16:21:57.921134 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:57.921101    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:21:57.926651 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:57.926623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:21:57.934096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:21:57.934069    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:25:06.010651 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.010615    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"]
Sep 11 16:25:06.011165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.010998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" containerName="console"
Sep 11 16:25:06.011165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.011011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" containerName="console"
Sep 11 16:25:06.011165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.011066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66a369d-0b5e-46ed-87bf-725ea9ccae54" containerName="console"
Sep 11 16:25:06.013944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.013929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.017310 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.017290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-xbmbn\""
Sep 11 16:25:06.017429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.017363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 11 16:25:06.017800 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.017780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 11 16:25:06.017800 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.017801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 11 16:25:06.023232 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.023210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.023329 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.023315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kdk\" (UniqueName: \"kubernetes.io/projected/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-kube-api-access-q6kdk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.028646 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.028626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"]
Sep 11 16:25:06.123963 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.123924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kdk\" (UniqueName: \"kubernetes.io/projected/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-kube-api-access-q6kdk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.124152 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.123973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.126260 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.126237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.140590 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.140570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kdk\" (UniqueName: \"kubernetes.io/projected/93d1fbb8-9ee1-47ac-aac4-19912989ca1d-kube-api-access-q6kdk\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fjzwz\" (UID: \"93d1fbb8-9ee1-47ac-aac4-19912989ca1d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.324562 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.324473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:06.605834 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.605774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"]
Sep 11 16:25:06.608330 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:25:06.608294    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d1fbb8_9ee1_47ac_aac4_19912989ca1d.slice/crio-e77cfe088cb7aea43408cf28b95507998aa30732101277028bd9095f7814c516 WatchSource:0}: Error finding container e77cfe088cb7aea43408cf28b95507998aa30732101277028bd9095f7814c516: Status 404 returned error can't find the container with id e77cfe088cb7aea43408cf28b95507998aa30732101277028bd9095f7814c516
Sep 11 16:25:06.610066 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.610043    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:25:06.966365 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.966335    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8468d98dcf-9z2sl"]
Sep 11 16:25:06.971050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.971029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:06.981835 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:06.981810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8468d98dcf-9z2sl"]
Sep 11 16:25:07.029531 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.029493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz" event={"ID":"93d1fbb8-9ee1-47ac-aac4-19912989ca1d","Type":"ContainerStarted","Data":"e77cfe088cb7aea43408cf28b95507998aa30732101277028bd9095f7814c516"}
Sep 11 16:25:07.030768 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-console-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.030856 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.030856 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55tf\" (UniqueName: \"kubernetes.io/projected/3071b390-6c46-4173-a1b5-5d4530736b42-kube-api-access-w55tf\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.030856 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-oauth-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.030955 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-service-ca\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.030955 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.030942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-trusted-ca-bundle\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.031017 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.031006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-oauth-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132211 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-console-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132211 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55tf\" (UniqueName: \"kubernetes.io/projected/3071b390-6c46-4173-a1b5-5d4530736b42-kube-api-access-w55tf\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-oauth-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-service-ca\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-trusted-ca-bundle\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.132448 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-oauth-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.133005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.132978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-oauth-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.133110 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.133069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-console-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.133178 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.133136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-service-ca\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.133261 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.133242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3071b390-6c46-4173-a1b5-5d4530736b42-trusted-ca-bundle\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.135199 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.135174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-oauth-config\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.135400 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.135379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3071b390-6c46-4173-a1b5-5d4530736b42-console-serving-cert\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.141673 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.141650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55tf\" (UniqueName: \"kubernetes.io/projected/3071b390-6c46-4173-a1b5-5d4530736b42-kube-api-access-w55tf\") pod \"console-8468d98dcf-9z2sl\" (UID: \"3071b390-6c46-4173-a1b5-5d4530736b42\") " pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.280957 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.280863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:07.555737 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:07.555369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8468d98dcf-9z2sl"]
Sep 11 16:25:07.559328 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:25:07.559290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3071b390_6c46_4173_a1b5_5d4530736b42.slice/crio-c72425f7c95eee195faa6a71ac74444dbdab7913c8ce53d5c1cbd8634a692447 WatchSource:0}: Error finding container c72425f7c95eee195faa6a71ac74444dbdab7913c8ce53d5c1cbd8634a692447: Status 404 returned error can't find the container with id c72425f7c95eee195faa6a71ac74444dbdab7913c8ce53d5c1cbd8634a692447
Sep 11 16:25:08.043428 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:08.043336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468d98dcf-9z2sl" event={"ID":"3071b390-6c46-4173-a1b5-5d4530736b42","Type":"ContainerStarted","Data":"88b5334b005488581bb87c9afc47d5b076e2e6d69a16617e8fb39c40d84e624b"}
Sep 11 16:25:08.043428 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:08.043395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8468d98dcf-9z2sl" event={"ID":"3071b390-6c46-4173-a1b5-5d4530736b42","Type":"ContainerStarted","Data":"c72425f7c95eee195faa6a71ac74444dbdab7913c8ce53d5c1cbd8634a692447"}
Sep 11 16:25:08.062849 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:08.062765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8468d98dcf-9z2sl" podStartSLOduration=2.06274272 podStartE2EDuration="2.06274272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:08.061132307 +0000 UTC m=+490.555032123" watchObservedRunningTime="2026-09-11 16:25:08.06274272 +0000 UTC m=+490.556642532"
Sep 11 16:25:11.050852 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:11.050815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz" event={"ID":"93d1fbb8-9ee1-47ac-aac4-19912989ca1d","Type":"ContainerStarted","Data":"a9169822947fba2b0c25d234e6fa184639cc08ab3fda739f9f369fdca071b81f"}
Sep 11 16:25:11.051232 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:11.050879    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:11.082043 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:11.081993    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz" podStartSLOduration=2.611287152 podStartE2EDuration="6.081979541s" totalImagesPullingTime="3.470692389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:11.080593889 +0000 UTC m=+493.574493711" watchObservedRunningTime="2026-09-11 16:25:11.081979541 +0000 UTC m=+493.575879352"
Sep 11 16:25:17.281698 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:17.281642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:17.282065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:17.281719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:17.286279 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:17.286253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:18.086408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:18.086380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8468d98dcf-9z2sl"
Sep 11 16:25:18.131598 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:18.131564    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:25:32.058769 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:32.058739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fjzwz"
Sep 11 16:25:43.158290 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.158210    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-764c4dc4f7-272zd" podUID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" containerName="console" containerID="cri-o://9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8" gracePeriod=15
Sep 11 16:25:43.434568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.434498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:25:43.539431 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539431 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539746 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539746 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kw8f\" (UniqueName: \"kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.539853 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle\") pod \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\" (UID: \"4a23301a-4605-4f5d-b045-ee7c0a1b3b25\") "
Sep 11 16:25:43.540022 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config" (OuterVolumeSpecName: "console-config") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:43.540022 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.539991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:43.540022 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.540009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:43.540342 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.540068    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-oauth-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.540342 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.540088    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.540454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.540336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:43.541639 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.541616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:43.541749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.541731    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f" (OuterVolumeSpecName: "kube-api-access-2kw8f") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "kube-api-access-2kw8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:43.541749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.541740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a23301a-4605-4f5d-b045-ee7c0a1b3b25" (UID: "4a23301a-4605-4f5d-b045-ee7c0a1b3b25"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:43.640843 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.640811    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-trusted-ca-bundle\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.640843 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.640839    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-serving-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.640843 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.640848    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-console-oauth-config\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.641068 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.640857    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-service-ca\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:43.641068 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:43.640866    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kw8f\" (UniqueName: \"kubernetes.io/projected/4a23301a-4605-4f5d-b045-ee7c0a1b3b25-kube-api-access-2kw8f\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:25:44.197918 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.197882    2563 generic.go:356] "Generic (PLEG): container finished" podID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" containerID="9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8" exitCode=0
Sep 11 16:25:44.198327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.197929    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-764c4dc4f7-272zd"
Sep 11 16:25:44.198327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.197946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764c4dc4f7-272zd" event={"ID":"4a23301a-4605-4f5d-b045-ee7c0a1b3b25","Type":"ContainerDied","Data":"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"}
Sep 11 16:25:44.198327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.197990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-764c4dc4f7-272zd" event={"ID":"4a23301a-4605-4f5d-b045-ee7c0a1b3b25","Type":"ContainerDied","Data":"66d555c96aa0fc7bb51ecebd777085e196073a49c3c2a6f7633aa9749c4bda33"}
Sep 11 16:25:44.198327 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.198011    2563 scope.go:117] "RemoveContainer" containerID="9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"
Sep 11 16:25:44.206409 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.206387    2563 scope.go:117] "RemoveContainer" containerID="9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"
Sep 11 16:25:44.206690 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:25:44.206656    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8\": container with ID starting with 9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8 not found: ID does not exist" containerID="9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"
Sep 11 16:25:44.206781 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.206700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8"} err="failed to get container status \"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8\": rpc error: code = NotFound desc = could not find container \"9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8\": container with ID starting with 9cce8bcf72e74ee0780009e54574f62a72521d3d9352697b4bf2ccd6b750edc8 not found: ID does not exist"
Sep 11 16:25:44.214142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.214118    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:25:44.216547 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:44.216527    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-764c4dc4f7-272zd"]
Sep 11 16:25:46.047318 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:25:46.047281    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" path="/var/lib/kubelet/pods/4a23301a-4605-4f5d-b045-ee7c0a1b3b25/volumes"
Sep 11 16:26:16.211768 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.211738    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"]
Sep 11 16:26:16.212168 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.212083    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" containerName="console"
Sep 11 16:26:16.212168 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.212096    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" containerName="console"
Sep 11 16:26:16.212168 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.212166    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a23301a-4605-4f5d-b045-ee7c0a1b3b25" containerName="console"
Sep 11 16:26:16.216309 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.216287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.219285 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.219265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pnmcc\""
Sep 11 16:26:16.219429 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.219269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 16:26:16.219841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.219823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:26:16.229691 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.229651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"]
Sep 11 16:26:16.284181 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.284089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-kube-api-access-rjlfx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.284181 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.284139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.384534 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.384493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-kube-api-access-rjlfx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.384749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.384559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.384945 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.384928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.393560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.393530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlfx\" (UniqueName: \"kubernetes.io/projected/5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf-kube-api-access-rjlfx\") pod \"cert-manager-operator-controller-manager-684f79c9b9-7tgr7\" (UID: \"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.526451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.526414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"
Sep 11 16:26:16.799904 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:16.799881    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7"]
Sep 11 16:26:16.801530 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:26:16.801492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f19eea7_c2ab_48d0_ab2a_7a6f2733dcaf.slice/crio-671fabc1ddd8006b3dd36ef04d5c5b72538ebefe9de36b94af118e7b3938e040 WatchSource:0}: Error finding container 671fabc1ddd8006b3dd36ef04d5c5b72538ebefe9de36b94af118e7b3938e040: Status 404 returned error can't find the container with id 671fabc1ddd8006b3dd36ef04d5c5b72538ebefe9de36b94af118e7b3938e040
Sep 11 16:26:17.336968 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:17.336932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7" event={"ID":"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf","Type":"ContainerStarted","Data":"671fabc1ddd8006b3dd36ef04d5c5b72538ebefe9de36b94af118e7b3938e040"}
Sep 11 16:26:20.354357 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:20.354312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7" event={"ID":"5f19eea7-c2ab-48d0-ab2a-7a6f2733dcaf","Type":"ContainerStarted","Data":"b1c5bdf7bf0f03d6b852ed3bd2b09a5c802aa2bb7ea9bd22554765d2e8134f7c"}
Sep 11 16:26:20.375752 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:20.375672    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-7tgr7" podStartSLOduration=1.632564541 podStartE2EDuration="4.375651185s" totalImagesPullingTime="2.743086644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:20.374974453 +0000 UTC m=+562.868874266" watchObservedRunningTime="2026-09-11 16:26:20.375651185 +0000 UTC m=+562.869550998"
Sep 11 16:26:38.641986 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.641949    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"]
Sep 11 16:26:38.645163 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.645146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.648015 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.647990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 16:26:38.648139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.648001    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 16:26:38.648139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.648051    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hs795\""
Sep 11 16:26:38.653471 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.653448    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"]
Sep 11 16:26:38.741149 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.741107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zln4w\" (UniqueName: \"kubernetes.io/projected/298c6a1f-53c1-4156-81a0-95641d0c2375-kube-api-access-zln4w\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.741337 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.741210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298c6a1f-53c1-4156-81a0-95641d0c2375-tmp\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.841584 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.841545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zln4w\" (UniqueName: \"kubernetes.io/projected/298c6a1f-53c1-4156-81a0-95641d0c2375-kube-api-access-zln4w\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.841781 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.841606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298c6a1f-53c1-4156-81a0-95641d0c2375-tmp\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.841991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.841966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/298c6a1f-53c1-4156-81a0-95641d0c2375-tmp\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.850163 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.850129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zln4w\" (UniqueName: \"kubernetes.io/projected/298c6a1f-53c1-4156-81a0-95641d0c2375-kube-api-access-zln4w\") pod \"openshift-lws-operator-fd8ccff4c-965zk\" (UID: \"298c6a1f-53c1-4156-81a0-95641d0c2375\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:38.955246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:38.955210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"
Sep 11 16:26:39.224755 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:39.224727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk"]
Sep 11 16:26:39.227189 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:26:39.227163    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod298c6a1f_53c1_4156_81a0_95641d0c2375.slice/crio-2b3547d7daa1526d7cd239eadbdcc74fdb9f6fce0ed3fc4cad77435ea2424dc7 WatchSource:0}: Error finding container 2b3547d7daa1526d7cd239eadbdcc74fdb9f6fce0ed3fc4cad77435ea2424dc7: Status 404 returned error can't find the container with id 2b3547d7daa1526d7cd239eadbdcc74fdb9f6fce0ed3fc4cad77435ea2424dc7
Sep 11 16:26:39.444568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:39.444494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk" event={"ID":"298c6a1f-53c1-4156-81a0-95641d0c2375","Type":"ContainerStarted","Data":"2b3547d7daa1526d7cd239eadbdcc74fdb9f6fce0ed3fc4cad77435ea2424dc7"}
Sep 11 16:26:41.187577 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.187539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7kz"]
Sep 11 16:26:41.191604 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.191581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.194552 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.194524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 16:26:41.195369 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.195348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6j2s5\""
Sep 11 16:26:41.195560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.195372    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 16:26:41.200988 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.200964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7kz"]
Sep 11 16:26:41.258754 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.258717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pknh\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-kube-api-access-6pknh\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.258931 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.258793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.360090 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.360036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pknh\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-kube-api-access-6pknh\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.360273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.360112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.369442 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.369406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.369601 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.369537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pknh\" (UniqueName: \"kubernetes.io/projected/6034c654-cae7-43d3-8a89-fd6f02791c5a-kube-api-access-6pknh\") pod \"cert-manager-55b4d954d5-kh7kz\" (UID: \"6034c654-cae7-43d3-8a89-fd6f02791c5a\") " pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.502996 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.502971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh7kz"
Sep 11 16:26:41.776884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:41.776856    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7kz"]
Sep 11 16:26:41.778360 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:26:41.778330    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6034c654_cae7_43d3_8a89_fd6f02791c5a.slice/crio-49adf507cd11837128f4ef8d9fd893c8ad313dc9a1a72ce280ea9da21b9ee317 WatchSource:0}: Error finding container 49adf507cd11837128f4ef8d9fd893c8ad313dc9a1a72ce280ea9da21b9ee317: Status 404 returned error can't find the container with id 49adf507cd11837128f4ef8d9fd893c8ad313dc9a1a72ce280ea9da21b9ee317
Sep 11 16:26:42.462095 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:42.462052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk" event={"ID":"298c6a1f-53c1-4156-81a0-95641d0c2375","Type":"ContainerStarted","Data":"7eacff761e3922c0eaa9805135efeae3f929c01546b99cbd9cc46b7a425267a9"}
Sep 11 16:26:42.463359 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:42.463335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh7kz" event={"ID":"6034c654-cae7-43d3-8a89-fd6f02791c5a","Type":"ContainerStarted","Data":"49adf507cd11837128f4ef8d9fd893c8ad313dc9a1a72ce280ea9da21b9ee317"}
Sep 11 16:26:42.479822 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:42.479775    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-965zk" podStartSLOduration=2.236367264 podStartE2EDuration="4.47975866s" totalImagesPullingTime="2.243391396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:42.478831167 +0000 UTC m=+584.972730977" watchObservedRunningTime="2026-09-11 16:26:42.47975866 +0000 UTC m=+584.973658471"
Sep 11 16:26:45.479611 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:45.479572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh7kz" event={"ID":"6034c654-cae7-43d3-8a89-fd6f02791c5a","Type":"ContainerStarted","Data":"c6ed289b1d50c7437ab86cc6f06ea2f5309810b893487299d3624c24849542f7"}
Sep 11 16:26:45.497043 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:45.496992    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kh7kz" podStartSLOduration=1.628235007 podStartE2EDuration="4.496975228s" totalImagesPullingTime="2.868740221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:45.495439 +0000 UTC m=+587.989338808" watchObservedRunningTime="2026-09-11 16:26:45.496975228 +0000 UTC m=+587.990875040"
Sep 11 16:26:52.605114 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.605076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"]
Sep 11 16:26:52.608259 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.608242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.612105 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.612073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 16:26:52.612105 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.612100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 16:26:52.612289 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.612164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lk58v\""
Sep 11 16:26:52.612289 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.612081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 16:26:52.617595 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.617575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"]
Sep 11 16:26:52.658328 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.658294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.658484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.658353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ab617c2-9173-429b-baa5-4d5ba3a57f84-manager-config\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.658484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.658407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwdq\" (UniqueName: \"kubernetes.io/projected/0ab617c2-9173-429b-baa5-4d5ba3a57f84-kube-api-access-qcwdq\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.658484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.658424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-metrics-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.759131 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.759103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-metrics-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.759323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.759146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.759323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.759199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ab617c2-9173-429b-baa5-4d5ba3a57f84-manager-config\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.759323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.759236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwdq\" (UniqueName: \"kubernetes.io/projected/0ab617c2-9173-429b-baa5-4d5ba3a57f84-kube-api-access-qcwdq\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.759907 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.759885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ab617c2-9173-429b-baa5-4d5ba3a57f84-manager-config\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.761592 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.761560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.761718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.761603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ab617c2-9173-429b-baa5-4d5ba3a57f84-metrics-cert\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.770145 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.770121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwdq\" (UniqueName: \"kubernetes.io/projected/0ab617c2-9173-429b-baa5-4d5ba3a57f84-kube-api-access-qcwdq\") pod \"lws-controller-manager-5956778578-5wmx6\" (UID: \"0ab617c2-9173-429b-baa5-4d5ba3a57f84\") " pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:52.918528 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:52.918445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:53.189115 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:53.189089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"]
Sep 11 16:26:53.191641 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:26:53.191616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab617c2_9173_429b_baa5_4d5ba3a57f84.slice/crio-d23f99abbaea75c6b83ffdd1a1195873479d6702d7aea6dc6340e58dd862196c WatchSource:0}: Error finding container d23f99abbaea75c6b83ffdd1a1195873479d6702d7aea6dc6340e58dd862196c: Status 404 returned error can't find the container with id d23f99abbaea75c6b83ffdd1a1195873479d6702d7aea6dc6340e58dd862196c
Sep 11 16:26:53.514527 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:53.514437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6" event={"ID":"0ab617c2-9173-429b-baa5-4d5ba3a57f84","Type":"ContainerStarted","Data":"d23f99abbaea75c6b83ffdd1a1195873479d6702d7aea6dc6340e58dd862196c"}
Sep 11 16:26:58.112312 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:58.112282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:26:58.112799 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:58.112411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:26:58.538279 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:58.538241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6" event={"ID":"0ab617c2-9173-429b-baa5-4d5ba3a57f84","Type":"ContainerStarted","Data":"dd1d436ab002b6e50818b1718f70e87c0c372151fa4a47d385177037a48af994"}
Sep 11 16:26:58.538463 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:58.538382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:26:58.555284 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:26:58.555230    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6" podStartSLOduration=1.5952629809999999 podStartE2EDuration="6.555213878s" totalImagesPullingTime="4.959950897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:26:58.554402975 +0000 UTC m=+601.048302787" watchObservedRunningTime="2026-09-11 16:26:58.555213878 +0000 UTC m=+601.049113691"
Sep 11 16:27:09.544746 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:09.544716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5956778578-5wmx6"
Sep 11 16:27:42.366798 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.366757    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"]
Sep 11 16:27:42.371488 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.371470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.374092 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.374072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 16:27:42.375440 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.375419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:27:42.375535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.375441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2r47n\""
Sep 11 16:27:42.375535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.375441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 16:27:42.375535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.375425    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:27:42.381693 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.381659    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"]
Sep 11 16:27:42.466588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.466550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.466777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.466601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zps\" (UniqueName: \"kubernetes.io/projected/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-kube-api-access-z2zps\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.466777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.466626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.567568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.567530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.567764 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.567573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zps\" (UniqueName: \"kubernetes.io/projected/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-kube-api-access-z2zps\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.567764 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.567595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.567764 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:27:42.567712    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 16:27:42.567888 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:27:42.567780    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert podName:efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab nodeName:}" failed. No retries permitted until 2026-09-11 16:27:43.067761973 +0000 UTC m=+645.561661763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-k7wvk" (UID: "efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab") : secret "plugin-serving-cert" not found
Sep 11 16:27:42.568310 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.568292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:42.578944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:42.578923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zps\" (UniqueName: \"kubernetes.io/projected/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-kube-api-access-z2zps\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:43.072394 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:43.072353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:43.072567 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:27:43.072510    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 16:27:43.072607 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:27:43.072572    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert podName:efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab nodeName:}" failed. No retries permitted until 2026-09-11 16:27:44.072558314 +0000 UTC m=+646.566458104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-k7wvk" (UID: "efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab") : secret "plugin-serving-cert" not found
Sep 11 16:27:44.082078 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:44.082035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:44.084468 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:44.084444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-k7wvk\" (UID: \"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:44.180731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:44.180657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"
Sep 11 16:27:44.452160 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:44.452135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk"]
Sep 11 16:27:44.454157 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:27:44.454132    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefe8bdb5_ecc7_41aa_ab7a_4d4f7c019dab.slice/crio-fee9c66795ede36478500165dd9c634283205d0d6ad8d4c7ddb767c09464440b WatchSource:0}: Error finding container fee9c66795ede36478500165dd9c634283205d0d6ad8d4c7ddb767c09464440b: Status 404 returned error can't find the container with id fee9c66795ede36478500165dd9c634283205d0d6ad8d4c7ddb767c09464440b
Sep 11 16:27:44.742182 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:44.742093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk" event={"ID":"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab","Type":"ContainerStarted","Data":"fee9c66795ede36478500165dd9c634283205d0d6ad8d4c7ddb767c09464440b"}
Sep 11 16:27:49.773200 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:49.773155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk" event={"ID":"efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab","Type":"ContainerStarted","Data":"75007ae0e722543f6eb3faf5bec9045592d8b8701138bc0492abb04c768ffe88"}
Sep 11 16:27:49.789274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:27:49.789225    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-k7wvk" podStartSLOduration=3.204002408 podStartE2EDuration="7.789209239s" totalImagesPullingTime="4.585206831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:27:49.78849611 +0000 UTC m=+652.282395947" watchObservedRunningTime="2026-09-11 16:27:49.789209239 +0000 UTC m=+652.283109050"
Sep 11 16:28:26.873358 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.873322    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:26.876510 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.876491    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:26.878830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.878814    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2m97x\""
Sep 11 16:28:26.884959 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.884930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:26.994777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.994737    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:26.997991 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:26.997968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:27.004908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.004884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:27.052841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.052806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwxw\" (UniqueName: \"kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw\") pod \"authorino-7bcfd8f9cd-qtzd4\" (UID: \"e7d7e90f-554d-4ca3-9f0e-f579349288e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:27.153652 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.153550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwxw\" (UniqueName: \"kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw\") pod \"authorino-7bcfd8f9cd-qtzd4\" (UID: \"e7d7e90f-554d-4ca3-9f0e-f579349288e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:27.153850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.153700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9l4v\" (UniqueName: \"kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v\") pod \"authorino-59c974bd48-ghjzh\" (UID: \"0beae4fa-4eec-4b9f-a217-08193be4b723\") " pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:27.162278 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.162248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwxw\" (UniqueName: \"kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw\") pod \"authorino-7bcfd8f9cd-qtzd4\" (UID: \"e7d7e90f-554d-4ca3-9f0e-f579349288e8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:27.185906 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.185875    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:27.254782 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.254749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9l4v\" (UniqueName: \"kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v\") pod \"authorino-59c974bd48-ghjzh\" (UID: \"0beae4fa-4eec-4b9f-a217-08193be4b723\") " pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:27.263780 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.263755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9l4v\" (UniqueName: \"kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v\") pod \"authorino-59c974bd48-ghjzh\" (UID: \"0beae4fa-4eec-4b9f-a217-08193be4b723\") " pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:27.307264 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.307228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:27.454895 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.454869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:27.457368 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:28:27.457343    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7d7e90f_554d_4ca3_9f0e_f579349288e8.slice/crio-427c571409e5dfab0792332047cd513f46a0a3821d85aab7561e4128e9357604 WatchSource:0}: Error finding container 427c571409e5dfab0792332047cd513f46a0a3821d85aab7561e4128e9357604: Status 404 returned error can't find the container with id 427c571409e5dfab0792332047cd513f46a0a3821d85aab7561e4128e9357604
Sep 11 16:28:27.578947 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.578925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:27.580866 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:28:27.580837    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0beae4fa_4eec_4b9f_a217_08193be4b723.slice/crio-25fb2070930c80381de9012e35b4cd30936e2db8109d4cdcdb3dcf61504a0afb WatchSource:0}: Error finding container 25fb2070930c80381de9012e35b4cd30936e2db8109d4cdcdb3dcf61504a0afb: Status 404 returned error can't find the container with id 25fb2070930c80381de9012e35b4cd30936e2db8109d4cdcdb3dcf61504a0afb
Sep 11 16:28:27.951187 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.951149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" event={"ID":"e7d7e90f-554d-4ca3-9f0e-f579349288e8","Type":"ContainerStarted","Data":"427c571409e5dfab0792332047cd513f46a0a3821d85aab7561e4128e9357604"}
Sep 11 16:28:27.952418 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:27.952393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ghjzh" event={"ID":"0beae4fa-4eec-4b9f-a217-08193be4b723","Type":"ContainerStarted","Data":"25fb2070930c80381de9012e35b4cd30936e2db8109d4cdcdb3dcf61504a0afb"}
Sep 11 16:28:29.962742 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:29.962699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" event={"ID":"e7d7e90f-554d-4ca3-9f0e-f579349288e8","Type":"ContainerStarted","Data":"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"}
Sep 11 16:28:29.964276 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:29.964248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ghjzh" event={"ID":"0beae4fa-4eec-4b9f-a217-08193be4b723","Type":"ContainerStarted","Data":"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"}
Sep 11 16:28:29.979809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:29.979767    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" podStartSLOduration=2.257495371 podStartE2EDuration="3.97975307s" totalImagesPullingTime="1.722257699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:29.978475812 +0000 UTC m=+692.472375623" watchObservedRunningTime="2026-09-11 16:28:29.97975307 +0000 UTC m=+692.473652881"
Sep 11 16:28:29.999875 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:29.999829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-ghjzh" podStartSLOduration=2.397268145 podStartE2EDuration="3.999812939s" totalImagesPullingTime="1.602544794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:29.999070519 +0000 UTC m=+692.492970331" watchObservedRunningTime="2026-09-11 16:28:29.999812939 +0000 UTC m=+692.493712749"
Sep 11 16:28:30.028558 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:30.028522    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:31.973127 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:31.973065    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" podUID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" containerName="authorino" containerID="cri-o://aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba" gracePeriod=30
Sep 11 16:28:32.225557 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.225496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:32.292943 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.292908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bwxw\" (UniqueName: \"kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw\") pod \"e7d7e90f-554d-4ca3-9f0e-f579349288e8\" (UID: \"e7d7e90f-554d-4ca3-9f0e-f579349288e8\") "
Sep 11 16:28:32.295027 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.294999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw" (OuterVolumeSpecName: "kube-api-access-6bwxw") pod "e7d7e90f-554d-4ca3-9f0e-f579349288e8" (UID: "e7d7e90f-554d-4ca3-9f0e-f579349288e8"). InnerVolumeSpecName "kube-api-access-6bwxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:32.393947 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.393905    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwxw\" (UniqueName: \"kubernetes.io/projected/e7d7e90f-554d-4ca3-9f0e-f579349288e8-kube-api-access-6bwxw\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:28:32.979345 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.979307    2563 generic.go:356] "Generic (PLEG): container finished" podID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" containerID="aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba" exitCode=0
Sep 11 16:28:32.979345 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.979344    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4"
Sep 11 16:28:32.979870 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.979352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" event={"ID":"e7d7e90f-554d-4ca3-9f0e-f579349288e8","Type":"ContainerDied","Data":"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"}
Sep 11 16:28:32.979870 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.979392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qtzd4" event={"ID":"e7d7e90f-554d-4ca3-9f0e-f579349288e8","Type":"ContainerDied","Data":"427c571409e5dfab0792332047cd513f46a0a3821d85aab7561e4128e9357604"}
Sep 11 16:28:32.979870 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.979409    2563 scope.go:117] "RemoveContainer" containerID="aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"
Sep 11 16:28:32.987898 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.987882    2563 scope.go:117] "RemoveContainer" containerID="aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"
Sep 11 16:28:32.988158 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:28:32.988141    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba\": container with ID starting with aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba not found: ID does not exist" containerID="aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"
Sep 11 16:28:32.988217 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:32.988169    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba"} err="failed to get container status \"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba\": rpc error: code = NotFound desc = could not find container \"aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba\": container with ID starting with aa8952dbd1c07c70c9691582cd50c129cdc8b0ca395a6b3a5a52e82d1157cdba not found: ID does not exist"
Sep 11 16:28:33.003405 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:33.003379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:33.007417 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:33.007398    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qtzd4"]
Sep 11 16:28:34.041368 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:34.041335    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" path="/var/lib/kubelet/pods/e7d7e90f-554d-4ca3-9f0e-f579349288e8/volumes"
Sep 11 16:28:48.335485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.335452    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-hqmfz"]
Sep 11 16:28:48.336005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.335784    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" containerName="authorino"
Sep 11 16:28:48.336005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.335796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" containerName="authorino"
Sep 11 16:28:48.336005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.335863    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d7e90f-554d-4ca3-9f0e-f579349288e8" containerName="authorino"
Sep 11 16:28:48.340053 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.340036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.342528 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.342504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 16:28:48.345499 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.345477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-hqmfz"]
Sep 11 16:28:48.423464 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.423431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1185209c-7e68-4df9-b217-0191af2b4afe-tls-cert\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.423633 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.423478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl298\" (UniqueName: \"kubernetes.io/projected/1185209c-7e68-4df9-b217-0191af2b4afe-kube-api-access-dl298\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.524222 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.524180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1185209c-7e68-4df9-b217-0191af2b4afe-tls-cert\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.524380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.524234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl298\" (UniqueName: \"kubernetes.io/projected/1185209c-7e68-4df9-b217-0191af2b4afe-kube-api-access-dl298\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.526601 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.526579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1185209c-7e68-4df9-b217-0191af2b4afe-tls-cert\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.533397 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.533376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl298\" (UniqueName: \"kubernetes.io/projected/1185209c-7e68-4df9-b217-0191af2b4afe-kube-api-access-dl298\") pod \"authorino-6d9898bd99-hqmfz\" (UID: \"1185209c-7e68-4df9-b217-0191af2b4afe\") " pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.649638 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.649545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-hqmfz"
Sep 11 16:28:48.919011 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:48.918983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-hqmfz"]
Sep 11 16:28:49.061854 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:49.061814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-hqmfz" event={"ID":"1185209c-7e68-4df9-b217-0191af2b4afe","Type":"ContainerStarted","Data":"73bd79732efc0c8ed559dfde2c40cd4c6bd79ae472f0783af469cdd0da8e5279"}
Sep 11 16:28:50.067537 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.067496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-hqmfz" event={"ID":"1185209c-7e68-4df9-b217-0191af2b4afe","Type":"ContainerStarted","Data":"fd9dae2a2c922f601c990370016e8c4b6a9680a8a63e8aa6b19e3d5a125d15cf"}
Sep 11 16:28:50.082832 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.082781    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-hqmfz" podStartSLOduration=1.5446393409999999 podStartE2EDuration="2.082763878s" totalImagesPullingTime="538.124537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:50.081977886 +0000 UTC m=+712.575877697" watchObservedRunningTime="2026-09-11 16:28:50.082763878 +0000 UTC m=+712.576663694"
Sep 11 16:28:50.107797 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.107767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:50.107968 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.107947    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-ghjzh" podUID="0beae4fa-4eec-4b9f-a217-08193be4b723" containerName="authorino" containerID="cri-o://df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97" gracePeriod=30
Sep 11 16:28:50.349952 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.349923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:50.441539 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.441504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9l4v\" (UniqueName: \"kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v\") pod \"0beae4fa-4eec-4b9f-a217-08193be4b723\" (UID: \"0beae4fa-4eec-4b9f-a217-08193be4b723\") "
Sep 11 16:28:50.443497 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.443473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v" (OuterVolumeSpecName: "kube-api-access-w9l4v") pod "0beae4fa-4eec-4b9f-a217-08193be4b723" (UID: "0beae4fa-4eec-4b9f-a217-08193be4b723"). InnerVolumeSpecName "kube-api-access-w9l4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:28:50.542842 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:50.542808    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9l4v\" (UniqueName: \"kubernetes.io/projected/0beae4fa-4eec-4b9f-a217-08193be4b723-kube-api-access-w9l4v\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:28:51.072865 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.072830    2563 generic.go:356] "Generic (PLEG): container finished" podID="0beae4fa-4eec-4b9f-a217-08193be4b723" containerID="df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97" exitCode=0
Sep 11 16:28:51.073340 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.072870    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ghjzh"
Sep 11 16:28:51.073340 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.072909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ghjzh" event={"ID":"0beae4fa-4eec-4b9f-a217-08193be4b723","Type":"ContainerDied","Data":"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"}
Sep 11 16:28:51.073340 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.072946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ghjzh" event={"ID":"0beae4fa-4eec-4b9f-a217-08193be4b723","Type":"ContainerDied","Data":"25fb2070930c80381de9012e35b4cd30936e2db8109d4cdcdb3dcf61504a0afb"}
Sep 11 16:28:51.073340 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.072963    2563 scope.go:117] "RemoveContainer" containerID="df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"
Sep 11 16:28:51.082005 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.081987    2563 scope.go:117] "RemoveContainer" containerID="df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"
Sep 11 16:28:51.082278 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:28:51.082259    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97\": container with ID starting with df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97 not found: ID does not exist" containerID="df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"
Sep 11 16:28:51.082339 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.082292    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97"} err="failed to get container status \"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97\": rpc error: code = NotFound desc = could not find container \"df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97\": container with ID starting with df93ae38cea63942837f0bc3f5bea892a744199c454d028f37725547c03e9a97 not found: ID does not exist"
Sep 11 16:28:51.094715 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.094658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:51.096558 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:51.096536    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ghjzh"]
Sep 11 16:28:52.042075 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:28:52.042022    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0beae4fa-4eec-4b9f-a217-08193be4b723" path="/var/lib/kubelet/pods/0beae4fa-4eec-4b9f-a217-08193be4b723/volumes"
Sep 11 16:29:19.120667 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.120636    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"]
Sep 11 16:29:19.121059 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.120981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0beae4fa-4eec-4b9f-a217-08193be4b723" containerName="authorino"
Sep 11 16:29:19.121059 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.120993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0beae4fa-4eec-4b9f-a217-08193be4b723" containerName="authorino"
Sep 11 16:29:19.121059 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.121042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0beae4fa-4eec-4b9f-a217-08193be4b723" containerName="authorino"
Sep 11 16:29:19.124815 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.124798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.127387 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.127358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 11 16:29:19.127933 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.127910    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 11 16:29:19.128031 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.127948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 11 16:29:19.128529 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.128504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-9swkt\""
Sep 11 16:29:19.143943 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.143918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"]
Sep 11 16:29:19.290988 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.290824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daec17f5-e749-4df7-a172-e19f14606f66-cert\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.290988 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.290880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drzv\" (UniqueName: \"kubernetes.io/projected/daec17f5-e749-4df7-a172-e19f14606f66-kube-api-access-8drzv\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.391503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.391416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daec17f5-e749-4df7-a172-e19f14606f66-cert\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.391503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.391477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drzv\" (UniqueName: \"kubernetes.io/projected/daec17f5-e749-4df7-a172-e19f14606f66-kube-api-access-8drzv\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.393987 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.393963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/daec17f5-e749-4df7-a172-e19f14606f66-cert\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.402102 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.402076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drzv\" (UniqueName: \"kubernetes.io/projected/daec17f5-e749-4df7-a172-e19f14606f66-kube-api-access-8drzv\") pod \"llmisvc-controller-manager-5fcc6bb6b7-mqxx6\" (UID: \"daec17f5-e749-4df7-a172-e19f14606f66\") " pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.441381 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.441351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:19.711318 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:19.711284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"]
Sep 11 16:29:19.715072 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:29:19.715045    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddaec17f5_e749_4df7_a172_e19f14606f66.slice/crio-3ff35b8699351ab65e6791fa07cbb9f548757bb3fec798550491e7d598d699df WatchSource:0}: Error finding container 3ff35b8699351ab65e6791fa07cbb9f548757bb3fec798550491e7d598d699df: Status 404 returned error can't find the container with id 3ff35b8699351ab65e6791fa07cbb9f548757bb3fec798550491e7d598d699df
Sep 11 16:29:20.208029 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:20.207998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6" event={"ID":"daec17f5-e749-4df7-a172-e19f14606f66","Type":"ContainerStarted","Data":"3ff35b8699351ab65e6791fa07cbb9f548757bb3fec798550491e7d598d699df"}
Sep 11 16:29:23.229560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:23.228931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6" event={"ID":"daec17f5-e749-4df7-a172-e19f14606f66","Type":"ContainerStarted","Data":"663b49fa4bcd15b4a7262f28e92ce4753cc7b46cf4996b24b0259262f2725ffc"}
Sep 11 16:29:23.230036 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:23.229641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:29:23.246617 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:23.246559    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6" podStartSLOduration=1.3481083 podStartE2EDuration="4.246543968s" totalImagesPullingTime="2.898435668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:23.244841348 +0000 UTC m=+745.738741160" watchObservedRunningTime="2026-09-11 16:29:23.246543968 +0000 UTC m=+745.740443779"
Sep 11 16:29:55.240775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:29:55.240739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-5fcc6bb6b7-mqxx6"
Sep 11 16:31:58.149902 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:31:58.149831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:31:58.149902 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:31:58.149896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:32:32.769053 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.769012    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:32:32.772521 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.772501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.776469 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.776423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 11 16:32:32.776594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.776477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 11 16:32:32.776594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.776501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 11 16:32:32.777078 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.777028    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-wp7q7\""
Sep 11 16:32:32.787136 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.787111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:32:32.803194 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.803303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pggd\" (UniqueName: \"kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.803303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.803303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.803402 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.803438 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.803413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904646 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pggd\" (UniqueName: \"kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.904861 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.904848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.905121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.905088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.905121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.905113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.905233 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.905176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.907015 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.906994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.907266 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.907249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:32.913034 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:32.913017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pggd\" (UniqueName: \"kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd\") pod \"fc-smoke-test-kserve-7f4687897-vcmmk\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:33.087497 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:33.087401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:33.360655 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:33.360631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:32:33.362700 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:32:33.362660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5902897f_a8a8_4106_8967_521748f135b8.slice/crio-d441181293c5f8073d0f3a82f6a1e1cdc96d73b76694184b169590dba50afa5a WatchSource:0}: Error finding container d441181293c5f8073d0f3a82f6a1e1cdc96d73b76694184b169590dba50afa5a: Status 404 returned error can't find the container with id d441181293c5f8073d0f3a82f6a1e1cdc96d73b76694184b169590dba50afa5a
Sep 11 16:32:33.364458 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:33.364440    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:32:34.096258 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:34.096212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" event={"ID":"5902897f-a8a8-4106-8967-521748f135b8","Type":"ContainerStarted","Data":"d441181293c5f8073d0f3a82f6a1e1cdc96d73b76694184b169590dba50afa5a"}
Sep 11 16:32:35.102565 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:35.102526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" event={"ID":"5902897f-a8a8-4106-8967-521748f135b8","Type":"ContainerStarted","Data":"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"}
Sep 11 16:32:35.121011 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:35.120965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" podStartSLOduration=1.6097635399999999 podStartE2EDuration="3.120949981s" totalImagesPullingTime="1.511186441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:35.119372899 +0000 UTC m=+937.613272722" watchObservedRunningTime="2026-09-11 16:32:35.120949981 +0000 UTC m=+937.614849850"
Sep 11 16:32:36.088449 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:36.088416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:43.088695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:43.088594    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:43.100734 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:43.100709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:32:43.151078 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:32:43.151055    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:33:14.740306 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:14.740264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:33:14.752993    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b" cmd=["/bin/sleep","15"]
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:14.753049    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" message="" err=<
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:33:14.753067    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" podUID="5902897f-a8a8-4106-8967-521748f135b8" containerName="main" containerID="cri-o://d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"
Sep 11 16:33:14.753167 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:14.753130    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" podUID="5902897f-a8a8-4106-8967-521748f135b8" containerName="main" containerID="cri-o://d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b" gracePeriod=60
Sep 11 16:33:15.109806 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.109772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:33:15.277614 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.277805 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.277805 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pggd\" (UniqueName: \"kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.277805 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.277805 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.278013 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277822    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir\") pod \"5902897f-a8a8-4106-8967-521748f135b8\" (UID: \"5902897f-a8a8-4106-8967-521748f135b8\") "
Sep 11 16:33:15.278013 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.277904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home" (OuterVolumeSpecName: "home") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:15.278112 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.278037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:15.278165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.278107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:15.278165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.278130    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:15.278165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.278149    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:15.279869 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.279846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm" (OuterVolumeSpecName: "dshm") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:33:15.279969 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.279877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:15.279969 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.279928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd" (OuterVolumeSpecName: "kube-api-access-2pggd") pod "5902897f-a8a8-4106-8967-521748f135b8" (UID: "5902897f-a8a8-4106-8967-521748f135b8"). InnerVolumeSpecName "kube-api-access-2pggd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:15.293268 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.293241    2563 generic.go:356] "Generic (PLEG): container finished" podID="5902897f-a8a8-4106-8967-521748f135b8" containerID="d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b" exitCode=0
Sep 11 16:33:15.293372 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.293289    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"
Sep 11 16:33:15.293372 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.293321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" event={"ID":"5902897f-a8a8-4106-8967-521748f135b8","Type":"ContainerDied","Data":"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"}
Sep 11 16:33:15.293451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.293374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk" event={"ID":"5902897f-a8a8-4106-8967-521748f135b8","Type":"ContainerDied","Data":"d441181293c5f8073d0f3a82f6a1e1cdc96d73b76694184b169590dba50afa5a"}
Sep 11 16:33:15.293451 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.293400    2563 scope.go:117] "RemoveContainer" containerID="d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"
Sep 11 16:33:15.302388 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.302373    2563 scope.go:117] "RemoveContainer" containerID="d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"
Sep 11 16:33:15.302645 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:33:15.302625    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b\": container with ID starting with d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b not found: ID does not exist" containerID="d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"
Sep 11 16:33:15.302719 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.302652    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b"} err="failed to get container status \"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b\": rpc error: code = NotFound desc = could not find container \"d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b\": container with ID starting with d9e9a6e5b425de04d8802f4e6cf54b255032fae68eb15909eae2bc0429b0cd4b not found: ID does not exist"
Sep 11 16:33:15.314235 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.314215    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:33:15.321262 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.321238    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vcmmk"]
Sep 11 16:33:15.379333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.379296    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:15.379333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.379324    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5902897f-a8a8-4106-8967-521748f135b8-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:15.379333 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.379335    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pggd\" (UniqueName: \"kubernetes.io/projected/5902897f-a8a8-4106-8967-521748f135b8-kube-api-access-2pggd\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:15.379549 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:15.379345    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5902897f-a8a8-4106-8967-521748f135b8-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:33:16.042210 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:33:16.042181    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5902897f-a8a8-4106-8967-521748f135b8" path="/var/lib/kubelet/pods/5902897f-a8a8-4106-8967-521748f135b8/volumes"
Sep 11 16:34:09.981388 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.981355    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:09.981897 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.981720    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5902897f-a8a8-4106-8967-521748f135b8" containerName="main"
Sep 11 16:34:09.981897 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.981735    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5902897f-a8a8-4106-8967-521748f135b8" containerName="main"
Sep 11 16:34:09.981897 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.981819    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5902897f-a8a8-4106-8967-521748f135b8" containerName="main"
Sep 11 16:34:09.986009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.985989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:09.988935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.988915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 11 16:34:09.990068 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.990015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 11 16:34:09.990156 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.990015    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 11 16:34:09.990156 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.990093    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-brd2q\""
Sep 11 16:34:09.994322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:09.994296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:10.045789 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.045944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.045944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp59\" (UniqueName: \"kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.045944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.045944 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.046077 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.045953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.146951 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.146919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147091 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.146964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147091 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.146983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp59\" (UniqueName: \"kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147091 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147091 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147091 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147379 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147444 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.147503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.147471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.149198 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.149178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.149485 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.149468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.156965 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.156934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp59\" (UniqueName: \"kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59\") pod \"fc-concurrency-test-kserve-b5d44467c-8mqqw\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.297210 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.297132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:10.567377 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:10.567296    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:10.571418 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:34:10.571382    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda333769b_d628_42c6_bde2_4245207289d5.slice/crio-19ced1079f47f4486dbf3e725f2109f8d0db774cdb018b6415628b556143fdae WatchSource:0}: Error finding container 19ced1079f47f4486dbf3e725f2109f8d0db774cdb018b6415628b556143fdae: Status 404 returned error can't find the container with id 19ced1079f47f4486dbf3e725f2109f8d0db774cdb018b6415628b556143fdae
Sep 11 16:34:11.558578 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:11.558539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" event={"ID":"a333769b-d628-42c6-bde2-4245207289d5","Type":"ContainerStarted","Data":"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"}
Sep 11 16:34:11.558962 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:11.558732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" event={"ID":"a333769b-d628-42c6-bde2-4245207289d5","Type":"ContainerStarted","Data":"19ced1079f47f4486dbf3e725f2109f8d0db774cdb018b6415628b556143fdae"}
Sep 11 16:34:11.575533 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:11.575490    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" podStartSLOduration=2.575474788 podStartE2EDuration="2.575474788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:34:11.574534045 +0000 UTC m=+1034.068433856" watchObservedRunningTime="2026-09-11 16:34:11.575474788 +0000 UTC m=+1034.069374599"
Sep 11 16:34:12.298245 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:12.298207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:20.298249 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:20.298211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:20.310851 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:20.310829    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:20.610888 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:20.610809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:49.564112 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.564079    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:49.575287 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:34:49.575252    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:34:49.575287 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:34:49.575287 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:34:49.575287 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9" cmd=["/bin/sleep","15"]
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.575306    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" message="" err=<
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:34:49.575319    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main" containerID="cri-o://f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"
Sep 11 16:34:49.575532 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.575354    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main" containerID="cri-o://f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9" gracePeriod=60
Sep 11 16:34:49.600365 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.600332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Sep 11 16:34:49.915143 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.915119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:49.991455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psp59\" (UniqueName: \"kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991628 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs\") pod \"a333769b-d628-42c6-bde2-4245207289d5\" (UID: \"a333769b-d628-42c6-bde2-4245207289d5\") "
Sep 11 16:34:49.991925 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home" (OuterVolumeSpecName: "home") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:34:49.991925 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:34:49.991925 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.991914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:34:49.993800 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.993771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:34:49.993800 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.993785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59" (OuterVolumeSpecName: "kube-api-access-psp59") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "kube-api-access-psp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:34:49.993941 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:49.993850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm" (OuterVolumeSpecName: "dshm") pod "a333769b-d628-42c6-bde2-4245207289d5" (UID: "a333769b-d628-42c6-bde2-4245207289d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092443    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092466    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092475    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psp59\" (UniqueName: \"kubernetes.io/projected/a333769b-d628-42c6-bde2-4245207289d5-kube-api-access-psp59\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092493    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a333769b-d628-42c6-bde2-4245207289d5-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.092512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.092504    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a333769b-d628-42c6-bde2-4245207289d5-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:34:50.742052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.742013    2563 generic.go:356] "Generic (PLEG): container finished" podID="a333769b-d628-42c6-bde2-4245207289d5" containerID="f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9" exitCode=0
Sep 11 16:34:50.742452 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.742067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"
Sep 11 16:34:50.742452 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.742103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" event={"ID":"a333769b-d628-42c6-bde2-4245207289d5","Type":"ContainerDied","Data":"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"}
Sep 11 16:34:50.742452 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.742163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw" event={"ID":"a333769b-d628-42c6-bde2-4245207289d5","Type":"ContainerDied","Data":"19ced1079f47f4486dbf3e725f2109f8d0db774cdb018b6415628b556143fdae"}
Sep 11 16:34:50.742452 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.742189    2563 scope.go:117] "RemoveContainer" containerID="f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"
Sep 11 16:34:50.751110 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.751094    2563 scope.go:117] "RemoveContainer" containerID="f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"
Sep 11 16:34:50.751377 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:34:50.751356    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9\": container with ID starting with f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9 not found: ID does not exist" containerID="f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"
Sep 11 16:34:50.751427 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.751384    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9"} err="failed to get container status \"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9\": rpc error: code = NotFound desc = could not find container \"f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9\": container with ID starting with f06c958ba6385b2c85e49944d89e4dd2a931975e18cbc4b0b9d9168f6a020ca9 not found: ID does not exist"
Sep 11 16:34:50.759667 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.759642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:50.763609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:50.763585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8mqqw"]
Sep 11 16:34:52.047084 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:34:52.047052    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a333769b-d628-42c6-bde2-4245207289d5" path="/var/lib/kubelet/pods/a333769b-d628-42c6-bde2-4245207289d5/volumes"
Sep 11 16:36:03.646274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.646237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:36:03.646752 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.646564    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main"
Sep 11 16:36:03.646752 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.646575    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main"
Sep 11 16:36:03.646752 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.646641    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a333769b-d628-42c6-bde2-4245207289d5" containerName="main"
Sep 11 16:36:03.649553 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.649536    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.652542 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.652517    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 11 16:36:03.653011 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.652997    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-st64d\""
Sep 11 16:36:03.654240 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.654219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 11 16:36:03.654362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.654236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 11 16:36:03.654362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.654273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 11 16:36:03.654362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.654236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:36:03.654362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.654339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-v968p\""
Sep 11 16:36:03.664494 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.664471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:36:03.815316 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815507 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjjcs\" (UniqueName: \"kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.815724 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.815572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916317 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916317 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916501 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916538 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916538 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916641 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjjcs\" (UniqueName: \"kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916641 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916641 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916641 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916877 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916877 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.916980 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.916929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.917084 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.917063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.918985 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.918955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.919329 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.919307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.924723 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.924700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjjcs\" (UniqueName: \"kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:03.960167 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:03.960144    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:36:04.236649 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:04.236613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:36:04.240477 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:36:04.240452    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49d0a71_d910_48b2_9875_b64a52e6b542.slice/crio-85fb3e0904bebf60f2186b28d7cffee71607fa9a45e219d1576e6ab5e3e48a50 WatchSource:0}: Error finding container 85fb3e0904bebf60f2186b28d7cffee71607fa9a45e219d1576e6ab5e3e48a50: Status 404 returned error can't find the container with id 85fb3e0904bebf60f2186b28d7cffee71607fa9a45e219d1576e6ab5e3e48a50
Sep 11 16:36:05.090566 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:05.090531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerStarted","Data":"85fb3e0904bebf60f2186b28d7cffee71607fa9a45e219d1576e6ab5e3e48a50"}
Sep 11 16:36:08.107901 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:08.107858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerStarted","Data":"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"}
Sep 11 16:36:11.124190 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:11.124154    2563 generic.go:356] "Generic (PLEG): container finished" podID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerID="0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400" exitCode=0
Sep 11 16:36:11.124700 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:11.124227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerDied","Data":"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"}
Sep 11 16:36:41.311306 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:41.311269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerStarted","Data":"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"}
Sep 11 16:36:41.334147 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:41.334099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.18986517 podStartE2EDuration="38.334084728s" totalImagesPullingTime="33.062910912s" totalInitContainerRuntime="2.081308646s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:36:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:36:41.332727869 +0000 UTC m=+1183.826627681" watchObservedRunningTime="2026-09-11 16:36:41.334084728 +0000 UTC m=+1183.827984540"
Sep 11 16:36:58.179257 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:58.179227    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:36:58.180872 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:36:58.180848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:38:28.618632 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:28.618594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:38:43.641302 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:43.641249    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="main" containerID="cri-o://61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779" gracePeriod=45
Sep 11 16:38:44.308469 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.308440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:38:44.329455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329496    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329905 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjjcs\" (UniqueName: \"kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329905 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329707    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329905 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm\") pod \"c49d0a71-d910-48b2-9875-b64a52e6b542\" (UID: \"c49d0a71-d910-48b2-9875-b64a52e6b542\") "
Sep 11 16:38:44.329905 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.329867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache" (OuterVolumeSpecName: "model-cache") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:44.330187 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.330028    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.330502 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.330385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:38:44.330828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.330801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home" (OuterVolumeSpecName: "home") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:44.332821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.332776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:38:44.332821 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.332778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs" (OuterVolumeSpecName: "kube-api-access-cjjcs") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "kube-api-access-cjjcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:38:44.333606 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.333574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm" (OuterVolumeSpecName: "dshm") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:44.348121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.348090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:44.352209 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.352187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c49d0a71-d910-48b2-9875-b64a52e6b542" (UID: "c49d0a71-d910-48b2-9875-b64a52e6b542"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:38:44.430403 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430340    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c49d0a71-d910-48b2-9875-b64a52e6b542-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430403 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430367    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430403 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430379    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49d0a71-d910-48b2-9875-b64a52e6b542-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430411    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430421    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjjcs\" (UniqueName: \"kubernetes.io/projected/c49d0a71-d910-48b2-9875-b64a52e6b542-kube-api-access-cjjcs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430437    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.430568 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.430447    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49d0a71-d910-48b2-9875-b64a52e6b542-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:38:44.937848 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.937814    2563 generic.go:356] "Generic (PLEG): container finished" podID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerID="61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779" exitCode=0
Sep 11 16:38:44.938302 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.937874    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 11 16:38:44.938302 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.937873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerDied","Data":"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"}
Sep 11 16:38:44.938302 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.938039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"c49d0a71-d910-48b2-9875-b64a52e6b542","Type":"ContainerDied","Data":"85fb3e0904bebf60f2186b28d7cffee71607fa9a45e219d1576e6ab5e3e48a50"}
Sep 11 16:38:44.938302 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.938085    2563 scope.go:117] "RemoveContainer" containerID="61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"
Sep 11 16:38:44.947646 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.947622    2563 scope.go:117] "RemoveContainer" containerID="0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"
Sep 11 16:38:44.955519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.955500    2563 scope.go:117] "RemoveContainer" containerID="61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"
Sep 11 16:38:44.955805 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:38:44.955787    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779\": container with ID starting with 61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779 not found: ID does not exist" containerID="61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"
Sep 11 16:38:44.955865 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.955815    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779"} err="failed to get container status \"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779\": rpc error: code = NotFound desc = could not find container \"61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779\": container with ID starting with 61c1876635dc52a600de6a275113d556326810cf0f2518b42361aeed7dfbb779 not found: ID does not exist"
Sep 11 16:38:44.955865 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.955831    2563 scope.go:117] "RemoveContainer" containerID="0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"
Sep 11 16:38:44.956112 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:38:44.956097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400\": container with ID starting with 0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400 not found: ID does not exist" containerID="0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"
Sep 11 16:38:44.956163 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.956115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400"} err="failed to get container status \"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400\": rpc error: code = NotFound desc = could not find container \"0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400\": container with ID starting with 0f6aed7529cad3be098b17c7f59972492a7a6e4e15b89f84ab0a63c1bfdbc400 not found: ID does not exist"
Sep 11 16:38:44.960321 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.960300    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:38:44.964073 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:44.964052    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 11 16:38:46.041560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:38:46.041526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" path="/var/lib/kubelet/pods/c49d0a71-d910-48b2-9875-b64a52e6b542/volumes"
Sep 11 16:39:23.892547 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892467    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:39:23.893003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892821    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="main"
Sep 11 16:39:23.893003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="main"
Sep 11 16:39:23.893003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="storage-initializer"
Sep 11 16:39:23.893003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892853    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="storage-initializer"
Sep 11 16:39:23.893003 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.892908    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49d0a71-d910-48b2-9875-b64a52e6b542" containerName="main"
Sep 11 16:39:23.896956 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.896940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:23.900637 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.900601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 11 16:39:23.900637 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.900628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 11 16:39:23.900869 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.900601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-z9cqr\""
Sep 11 16:39:23.900869 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.900648    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 11 16:39:23.905766 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:23.905745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:39:24.052248 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.052410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.052410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.052410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbxv\" (UniqueName: \"kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.052529 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.052529 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.052444    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153085 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153085 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbxv\" (UniqueName: \"kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153487 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153487 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.153654 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.153619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.155288 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.155263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.155663 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.155643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.161154 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.161131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbxv\" (UniqueName: \"kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-4fjg5\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.207718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.207687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:24.478889 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:24.478864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:39:24.481376 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:39:24.481345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b579a6_a6c7_4524_8505_09af14fa80c3.slice/crio-5497b1683cefac48c3b40d0daa53f71f322e9d07a66c6ee0e92917f25174fa6c WatchSource:0}: Error finding container 5497b1683cefac48c3b40d0daa53f71f322e9d07a66c6ee0e92917f25174fa6c: Status 404 returned error can't find the container with id 5497b1683cefac48c3b40d0daa53f71f322e9d07a66c6ee0e92917f25174fa6c
Sep 11 16:39:25.129744 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:25.129701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" event={"ID":"e2b579a6-a6c7-4524-8505-09af14fa80c3","Type":"ContainerStarted","Data":"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"}
Sep 11 16:39:25.130222 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:25.129765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" event={"ID":"e2b579a6-a6c7-4524-8505-09af14fa80c3","Type":"ContainerStarted","Data":"5497b1683cefac48c3b40d0daa53f71f322e9d07a66c6ee0e92917f25174fa6c"}
Sep 11 16:39:25.149830 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:25.149776    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" podStartSLOduration=2.149759017 podStartE2EDuration="2.149759017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:39:25.148538979 +0000 UTC m=+1347.642438802" watchObservedRunningTime="2026-09-11 16:39:25.149759017 +0000 UTC m=+1347.643658828"
Sep 11 16:39:25.208142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:25.208106    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:34.208524 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:34.208487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:34.222424 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:34.222390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:35.187742 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:35.187711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:57.709666 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:57.709626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:39:57.720378 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:39:57.720330    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:39:57.720378 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:39:57.720378 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:39:57.720378 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4" cmd=["/bin/sleep","15"]
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:57.720402    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" message="" err=<
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:39:57.720419    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerName="main" containerID="cri-o://62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"
Sep 11 16:39:57.720726 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:57.720465    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerName="main" containerID="cri-o://62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4" gracePeriod=60
Sep 11 16:39:58.059932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.059903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:58.143545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlbxv\" (UniqueName: \"kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.143739 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.143739 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.143739 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143597    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.143739 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143616    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.143739 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs\") pod \"e2b579a6-a6c7-4524-8505-09af14fa80c3\" (UID: \"e2b579a6-a6c7-4524-8505-09af14fa80c3\") "
Sep 11 16:39:58.144009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:58.144009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.143915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home" (OuterVolumeSpecName: "home") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:58.144162 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.144126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:58.145796 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.145773    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm" (OuterVolumeSpecName: "dshm") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:39:58.145908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.145856    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:39:58.145908 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.145885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv" (OuterVolumeSpecName: "kube-api-access-tlbxv") pod "e2b579a6-a6c7-4524-8505-09af14fa80c3" (UID: "e2b579a6-a6c7-4524-8505-09af14fa80c3"). InnerVolumeSpecName "kube-api-access-tlbxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245177    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245202    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245211    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245218    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b579a6-a6c7-4524-8505-09af14fa80c3-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245226    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b579a6-a6c7-4524-8505-09af14fa80c3-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.245246 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.245235    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbxv\" (UniqueName: \"kubernetes.io/projected/e2b579a6-a6c7-4524-8505-09af14fa80c3-kube-api-access-tlbxv\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:39:58.289603 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.289565    2563 generic.go:356] "Generic (PLEG): container finished" podID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerID="62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4" exitCode=0
Sep 11 16:39:58.289812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.289608    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"
Sep 11 16:39:58.289812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.289655    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" event={"ID":"e2b579a6-a6c7-4524-8505-09af14fa80c3","Type":"ContainerDied","Data":"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"}
Sep 11 16:39:58.289812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.289723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5" event={"ID":"e2b579a6-a6c7-4524-8505-09af14fa80c3","Type":"ContainerDied","Data":"5497b1683cefac48c3b40d0daa53f71f322e9d07a66c6ee0e92917f25174fa6c"}
Sep 11 16:39:58.289812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.289753    2563 scope.go:117] "RemoveContainer" containerID="62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"
Sep 11 16:39:58.298955 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.298941    2563 scope.go:117] "RemoveContainer" containerID="62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"
Sep 11 16:39:58.299238 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:39:58.299218    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4\": container with ID starting with 62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4 not found: ID does not exist" containerID="62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"
Sep 11 16:39:58.299280 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.299246    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4"} err="failed to get container status \"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4\": rpc error: code = NotFound desc = could not find container \"62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4\": container with ID starting with 62869c27d844df2441fee10e8c1f387fbc5581ab25b1829cbd19c0b0869692a4 not found: ID does not exist"
Sep 11 16:39:58.311792 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.311769    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:39:58.315579 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:39:58.315558    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-4fjg5"]
Sep 11 16:40:00.041774 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:00.041726    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" path="/var/lib/kubelet/pods/e2b579a6-a6c7-4524-8505-09af14fa80c3/volumes"
Sep 11 16:40:43.290899 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.290867    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:40:43.291335 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.291275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerName="main"
Sep 11 16:40:43.291335 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.291290    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerName="main"
Sep 11 16:40:43.291405 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.291365    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b579a6-a6c7-4524-8505-09af14fa80c3" containerName="main"
Sep 11 16:40:43.294487 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.294470    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.297183 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.297124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 11 16:40:43.297363 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.297346    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:40:43.297492 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.297265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 11 16:40:43.297601 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.297208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 11 16:40:43.297601 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.297365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 11 16:40:43.298125 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.298105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-s4rx9\""
Sep 11 16:40:43.304776 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.304748    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:40:43.328118 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s85t\" (UniqueName: \"kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.328380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.328329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429309 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429309 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s85t\" (UniqueName: \"kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429975 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429975 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.429975 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.429880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.430085 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.430018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.430291 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.430267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.431812 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.431792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.432228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.432212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.438035 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.438014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s85t\" (UniqueName: \"kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t\") pod \"auth-invalid-token-test-kserve-55bfd7f5fb-x42lt\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.606242 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.606157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:43.887166 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:43.887139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:40:43.888932 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:40:43.888894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae9949_238f_44b7_9277_cb1d8ea22d5f.slice/crio-5c0ef8b5267a3cbf0136717a60496470ac7dd3dbb773d758a391f3d475e01186 WatchSource:0}: Error finding container 5c0ef8b5267a3cbf0136717a60496470ac7dd3dbb773d758a391f3d475e01186: Status 404 returned error can't find the container with id 5c0ef8b5267a3cbf0136717a60496470ac7dd3dbb773d758a391f3d475e01186
Sep 11 16:40:44.504275 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:44.504241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerStarted","Data":"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"}
Sep 11 16:40:44.504645 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:44.504292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerStarted","Data":"5c0ef8b5267a3cbf0136717a60496470ac7dd3dbb773d758a391f3d475e01186"}
Sep 11 16:40:46.515210 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:46.515172    2563 generic.go:356] "Generic (PLEG): container finished" podID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerID="bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4" exitCode=0
Sep 11 16:40:46.515587 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:46.515245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerDied","Data":"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"}
Sep 11 16:40:47.522080 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:47.522041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerStarted","Data":"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"}
Sep 11 16:40:47.546314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:47.546230    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podStartSLOduration=2.326229675 podStartE2EDuration="4.546212351s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.219982676s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:40:47.54499516 +0000 UTC m=+1430.038894975" watchObservedRunningTime="2026-09-11 16:40:47.546212351 +0000 UTC m=+1430.040112163"
Sep 11 16:40:47.606419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:47.606370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:53.607158 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:53.607067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:40:53.608591 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:40:53.608561    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:03.607415 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:03.607366    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:13.607450 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:13.607411    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:23.607311 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:23.607263    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:33.607043 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:33.606987    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:43.607433 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:43.607385    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:53.606863 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:53.606821    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:41:58.210712 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:58.210662    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:41:58.213070 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:41:58.213043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:42:03.607051 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:03.607003    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:42:13.607252 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:13.607203    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:42:23.607654 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:23.607534    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:42:33.607306 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:33.607251    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:42:43.617367 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:43.617328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:42:43.627512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:42:43.627477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:43:04.958162 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:04.958127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:43:19.980578 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:19.980534    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" containerID="cri-o://c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156" gracePeriod=45
Sep 11 16:43:20.617959 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.617914    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 11 16:43:20.733410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.733383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s85t\" (UniqueName: \"kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.819995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.820047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.820713 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.820075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache\") pod \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\" (UID: \"e2ae9949-238f-44b7-9277-cb1d8ea22d5f\") "
Sep 11 16:43:20.821595 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.821256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:20.824349 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.824165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:43:20.825170 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.824865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home" (OuterVolumeSpecName: "home") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:20.826137 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.826106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:43:20.827319 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.827293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t" (OuterVolumeSpecName: "kube-api-access-8s85t") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "kube-api-access-8s85t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:43:20.828020 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.827999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm" (OuterVolumeSpecName: "dshm") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:20.837438 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.837415    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:20.854806 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.854776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2ae9949-238f-44b7-9277-cb1d8ea22d5f" (UID: "e2ae9949-238f-44b7-9277-cb1d8ea22d5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:43:20.921609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s85t\" (UniqueName: \"kubernetes.io/projected/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kube-api-access-8s85t\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921609    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921620    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921630    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921639    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921646    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921655    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:20.921779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:20.921662    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2ae9949-238f-44b7-9277-cb1d8ea22d5f-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:43:21.286829 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.286790    2563 generic.go:356] "Generic (PLEG): container finished" podID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerID="c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156" exitCode=0
Sep 11 16:43:21.287228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.286863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"
Sep 11 16:43:21.287228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.286875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerDied","Data":"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"}
Sep 11 16:43:21.287228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.286944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt" event={"ID":"e2ae9949-238f-44b7-9277-cb1d8ea22d5f","Type":"ContainerDied","Data":"5c0ef8b5267a3cbf0136717a60496470ac7dd3dbb773d758a391f3d475e01186"}
Sep 11 16:43:21.287228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.286968    2563 scope.go:117] "RemoveContainer" containerID="c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"
Sep 11 16:43:21.295797 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.295781    2563 scope.go:117] "RemoveContainer" containerID="bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"
Sep 11 16:43:21.303445 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.303427    2563 scope.go:117] "RemoveContainer" containerID="c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"
Sep 11 16:43:21.303721 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:43:21.303699    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156\": container with ID starting with c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156 not found: ID does not exist" containerID="c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"
Sep 11 16:43:21.303777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.303730    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156"} err="failed to get container status \"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156\": rpc error: code = NotFound desc = could not find container \"c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156\": container with ID starting with c319a7b85508b481299f0cc5f5e208bc71b10d86f046160cec65831a5cdc7156 not found: ID does not exist"
Sep 11 16:43:21.303777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.303746    2563 scope.go:117] "RemoveContainer" containerID="bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"
Sep 11 16:43:21.303968 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:43:21.303950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4\": container with ID starting with bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4 not found: ID does not exist" containerID="bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"
Sep 11 16:43:21.304024 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.303978    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4"} err="failed to get container status \"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4\": rpc error: code = NotFound desc = could not find container \"bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4\": container with ID starting with bc82fe9cc1b0594ead9366be80d2fdc88dd9e108e52a883da4655d200dc6f0c4 not found: ID does not exist"
Sep 11 16:43:21.309922 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.309900    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:43:21.313729 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:21.313704    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-55bfd7f5fb-x42lt"]
Sep 11 16:43:22.043111 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:43:22.043076    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" path="/var/lib/kubelet/pods/e2ae9949-238f-44b7-9277-cb1d8ea22d5f/volumes"
Sep 11 16:44:20.081789 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.081746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:44:20.082254 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.082238    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="storage-initializer"
Sep 11 16:44:20.082304 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.082258    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="storage-initializer"
Sep 11 16:44:20.082304 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.082270    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main"
Sep 11 16:44:20.082304 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.082279    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main"
Sep 11 16:44:20.082408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.082374    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ae9949-238f-44b7-9277-cb1d8ea22d5f" containerName="main"
Sep 11 16:44:20.085930 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.085911    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.088647 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.088616    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 11 16:44:20.088779 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.088670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-7wlxl\""
Sep 11 16:44:20.089511 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.089486    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:44:20.089849 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.089825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 11 16:44:20.089958 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.089825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 11 16:44:20.090215 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.090197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 11 16:44:20.098826 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.098804    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:44:20.120287 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120417 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120417 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120417 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.120588 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.120550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxsd\" (UniqueName: \"kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221220 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxsd\" (UniqueName: \"kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221408 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221882 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.221882 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.222041 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.222041 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.221986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.222204 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.222188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.223705 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.223662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.223928 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.223909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.230319 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.230300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxsd\" (UniqueName: \"kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd\") pod \"auth-disabled-test-kserve-74487f7767-hf8x9\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.396656 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.396560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:20.676884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:20.676858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:44:20.679225 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:44:20.679190    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68dce28_d9e9_45c8_b8f1_d55840ebbaaa.slice/crio-7caab06c885bb71a26170d23501509176558e1c08a2781cdb526fb7c07c607c0 WatchSource:0}: Error finding container 7caab06c885bb71a26170d23501509176558e1c08a2781cdb526fb7c07c607c0: Status 404 returned error can't find the container with id 7caab06c885bb71a26170d23501509176558e1c08a2781cdb526fb7c07c607c0
Sep 11 16:44:21.575582 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:21.575541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerStarted","Data":"ca63a457606258bd5992a5fdb7aba268018add7a66c085d2bbd44d5207090360"}
Sep 11 16:44:21.575997 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:21.575609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerStarted","Data":"7caab06c885bb71a26170d23501509176558e1c08a2781cdb526fb7c07c607c0"}
Sep 11 16:44:23.587851 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:23.587814    2563 generic.go:356] "Generic (PLEG): container finished" podID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerID="ca63a457606258bd5992a5fdb7aba268018add7a66c085d2bbd44d5207090360" exitCode=0
Sep 11 16:44:23.588223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:23.587888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerDied","Data":"ca63a457606258bd5992a5fdb7aba268018add7a66c085d2bbd44d5207090360"}
Sep 11 16:44:24.596120 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:24.596083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerStarted","Data":"7beaef799258b2bd31829def7f728dd631cccd72259ecdd3794a5dd067305216"}
Sep 11 16:44:24.618328 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:24.618279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podStartSLOduration=2.420941345 podStartE2EDuration="4.618264435s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.19732309s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:24.617142093 +0000 UTC m=+1647.111041904" watchObservedRunningTime="2026-09-11 16:44:24.618264435 +0000 UTC m=+1647.112164248"
Sep 11 16:44:25.397661 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:25.397624    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:30.397525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:30.397487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:44:30.399154 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:30.399124    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:44:35.013096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.013057    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:44:35.016438 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.016413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.019258 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.019220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 11 16:44:35.020618 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.020321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 11 16:44:35.020618 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.020432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 11 16:44:35.020618 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.020552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-f7v8n\""
Sep 11 16:44:35.020618 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.020591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-p7nmf\""
Sep 11 16:44:35.026303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.026238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:44:35.159054 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.159019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.159224 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.159068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzr5\" (UniqueName: \"kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.259883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.259843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.259883 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.259888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzr5\" (UniqueName: \"kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.262589 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.262561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.269175 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.269107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzr5\" (UniqueName: \"kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.331411 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.331377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:35.605274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.605240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:44:35.608109 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:44:35.608078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca31e6_2d9f_4604_a0c3_8e0eef182287.slice/crio-0a01d3ac7070008336cb832a6a22d6c81539172f97703e2765611cf6e92ce4e4 WatchSource:0}: Error finding container 0a01d3ac7070008336cb832a6a22d6c81539172f97703e2765611cf6e92ce4e4: Status 404 returned error can't find the container with id 0a01d3ac7070008336cb832a6a22d6c81539172f97703e2765611cf6e92ce4e4
Sep 11 16:44:35.610614 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.610583    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:44:35.656635 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:35.656592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" event={"ID":"f6ca31e6-2d9f-4604-a0c3-8e0eef182287","Type":"ContainerStarted","Data":"0a01d3ac7070008336cb832a6a22d6c81539172f97703e2765611cf6e92ce4e4"}
Sep 11 16:44:38.674608 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:38.674567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" event={"ID":"f6ca31e6-2d9f-4604-a0c3-8e0eef182287","Type":"ContainerStarted","Data":"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"}
Sep 11 16:44:38.675014 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:38.674721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:44:38.691842 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:38.691788    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podStartSLOduration=2.201089452 podStartE2EDuration="4.691768297s" totalImagesPullingTime="2.490678845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:44:38.691214912 +0000 UTC m=+1661.185114714" watchObservedRunningTime="2026-09-11 16:44:38.691768297 +0000 UTC m=+1661.185668109"
Sep 11 16:44:40.397484 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:40.397442    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:44:50.397262 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:44:50.397216    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:00.397040 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:00.396998    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:09.681762 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:09.681721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:10.397187 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:10.397130    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:11.116810 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:11.116774    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:45:11.118640 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:11.118603    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:20.397545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:20.397440    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:21.122704 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:21.119339    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:26.139096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:26.139025    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" containerID="cri-o://c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3" gracePeriod=45
Sep 11 16:45:30.397089 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:30.397036    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:31.119040 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:31.118992    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:40.397322 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:40.397273    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:41.119251 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:41.119211    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:50.397850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:50.397800    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:45:51.118668 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:51.118628    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:45:56.392203 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.392179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:45:56.483598 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.483563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqzr5\" (UniqueName: \"kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5\") pod \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") "
Sep 11 16:45:56.483783 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.483607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs\") pod \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\" (UID: \"f6ca31e6-2d9f-4604-a0c3-8e0eef182287\") "
Sep 11 16:45:56.485851 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.485820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6ca31e6-2d9f-4604-a0c3-8e0eef182287" (UID: "f6ca31e6-2d9f-4604-a0c3-8e0eef182287"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:45:56.485958 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.485831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5" (OuterVolumeSpecName: "kube-api-access-zqzr5") pod "f6ca31e6-2d9f-4604-a0c3-8e0eef182287" (UID: "f6ca31e6-2d9f-4604-a0c3-8e0eef182287"). InnerVolumeSpecName "kube-api-access-zqzr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:45:56.585236 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.585194    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzr5\" (UniqueName: \"kubernetes.io/projected/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-kube-api-access-zqzr5\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:45:56.585236 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:56.585234    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6ca31e6-2d9f-4604-a0c3-8e0eef182287-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:45:57.068689 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.068644    2563 generic.go:356] "Generic (PLEG): container finished" podID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerID="c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3" exitCode=0
Sep 11 16:45:57.068858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.068704    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"
Sep 11 16:45:57.068858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.068713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" event={"ID":"f6ca31e6-2d9f-4604-a0c3-8e0eef182287","Type":"ContainerDied","Data":"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"}
Sep 11 16:45:57.068858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.068775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5" event={"ID":"f6ca31e6-2d9f-4604-a0c3-8e0eef182287","Type":"ContainerDied","Data":"0a01d3ac7070008336cb832a6a22d6c81539172f97703e2765611cf6e92ce4e4"}
Sep 11 16:45:57.068858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.068805    2563 scope.go:117] "RemoveContainer" containerID="c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"
Sep 11 16:45:57.079043 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.079018    2563 scope.go:117] "RemoveContainer" containerID="c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"
Sep 11 16:45:57.079314 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:45:57.079296    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3\": container with ID starting with c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3 not found: ID does not exist" containerID="c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"
Sep 11 16:45:57.079384 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.079329    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3"} err="failed to get container status \"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3\": rpc error: code = NotFound desc = could not find container \"c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3\": container with ID starting with c6cfe111620f7c75687d5f200431a3130f8379061baaae043a8dd8baf21e5af3 not found: ID does not exist"
Sep 11 16:45:57.090900 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.090876    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:45:57.096620 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:57.096598    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5f6gk5"]
Sep 11 16:45:58.043237 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:45:58.043206    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" path="/var/lib/kubelet/pods/f6ca31e6-2d9f-4604-a0c3-8e0eef182287/volumes"
Sep 11 16:46:00.397278 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:00.397234    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:46:10.397245 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:10.397201    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 11 16:46:20.407093 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:20.407057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:46:20.414791 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:20.414765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:46:37.647380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:37.647338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:46:52.669218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:52.669105    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" containerID="cri-o://7beaef799258b2bd31829def7f728dd631cccd72259ecdd3794a5dd067305216" gracePeriod=45
Sep 11 16:46:53.354416 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.354382    2563 generic.go:356] "Generic (PLEG): container finished" podID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerID="7beaef799258b2bd31829def7f728dd631cccd72259ecdd3794a5dd067305216" exitCode=0
Sep 11 16:46:53.354582 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.354519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerDied","Data":"7beaef799258b2bd31829def7f728dd631cccd72259ecdd3794a5dd067305216"}
Sep 11 16:46:53.419525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.419501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:46:53.594489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594489 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhxsd\" (UniqueName: \"kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.594772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.594758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache\") pod \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\" (UID: \"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa\") "
Sep 11 16:46:53.595250 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.595217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:53.595363 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.595268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home" (OuterVolumeSpecName: "home") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:53.595760 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.595737    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:46:53.596967 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.596937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:46:53.597074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.597056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd" (OuterVolumeSpecName: "kube-api-access-jhxsd") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "kube-api-access-jhxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:46:53.597127 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.597079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm" (OuterVolumeSpecName: "dshm") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:53.613512 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.613483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:53.625784 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.625667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" (UID: "c68dce28-d9e9-45c8-b8f1-d55840ebbaaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:46:53.695793 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695762    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhxsd\" (UniqueName: \"kubernetes.io/projected/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kube-api-access-jhxsd\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.695793 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695791    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695802    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695812    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695821    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695829    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695837    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:53.696121 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:53.695844    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:46:54.360850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.360774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"
Sep 11 16:46:54.360850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.360794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" event={"ID":"c68dce28-d9e9-45c8-b8f1-d55840ebbaaa","Type":"ContainerDied","Data":"7caab06c885bb71a26170d23501509176558e1c08a2781cdb526fb7c07c607c0"}
Sep 11 16:46:54.360850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.360850    2563 scope.go:117] "RemoveContainer" containerID="7beaef799258b2bd31829def7f728dd631cccd72259ecdd3794a5dd067305216"
Sep 11 16:46:54.369371 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.369354    2563 scope.go:117] "RemoveContainer" containerID="ca63a457606258bd5992a5fdb7aba268018add7a66c085d2bbd44d5207090360"
Sep 11 16:46:54.379828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.379808    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:46:54.383970 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.383950    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9"]
Sep 11 16:46:54.408383 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:54.408348    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-74487f7767-hf8x9" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 16:46:56.042737 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:56.042708    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" path="/var/lib/kubelet/pods/c68dce28-d9e9-45c8-b8f1-d55840ebbaaa/volumes"
Sep 11 16:46:58.240491 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:58.240464    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:46:58.244711 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:46:58.244688    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:48:08.225591 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225558    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225907    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225919    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225926    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225932    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225954    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="storage-initializer"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.225960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="storage-initializer"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.226021    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ca31e6-2d9f-4604-a0c3-8e0eef182287" containerName="main"
Sep 11 16:48:08.227422 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.226031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68dce28-d9e9-45c8-b8f1-d55840ebbaaa" containerName="main"
Sep 11 16:48:08.228796 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.228781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.232447 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.232421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 11 16:48:08.232587 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.232480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-tnj6n\""
Sep 11 16:48:08.232587 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.232520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-6vttl\""
Sep 11 16:48:08.232721 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.232670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 11 16:48:08.232784 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.232742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 11 16:48:08.237490 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.237469    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:48:08.314593 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.314551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.314806 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.314655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6zg\" (UniqueName: \"kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.415188 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.415148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6zg\" (UniqueName: \"kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.415367 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.415217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.417661 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.417627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.424146 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.424122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6zg\" (UniqueName: \"kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.540555 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.540467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:08.814468 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:08.814338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:48:08.817545 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:48:08.817511    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86538e8_4180_4f71_b64c_b9c05a02bffa.slice/crio-957e395e4fc0b0a9701d4034c9db2c516b91d8e887e258877177a9d34d85eb7b WatchSource:0}: Error finding container 957e395e4fc0b0a9701d4034c9db2c516b91d8e887e258877177a9d34d85eb7b: Status 404 returned error can't find the container with id 957e395e4fc0b0a9701d4034c9db2c516b91d8e887e258877177a9d34d85eb7b
Sep 11 16:48:09.712143 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:09.712104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" event={"ID":"d86538e8-4180-4f71-b64c-b9c05a02bffa","Type":"ContainerStarted","Data":"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"}
Sep 11 16:48:09.712534 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:09.712158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" event={"ID":"d86538e8-4180-4f71-b64c-b9c05a02bffa","Type":"ContainerStarted","Data":"957e395e4fc0b0a9701d4034c9db2c516b91d8e887e258877177a9d34d85eb7b"}
Sep 11 16:48:09.712534 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:09.712237    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:48:09.729061 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:09.729012    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podStartSLOduration=1.728997606 podStartE2EDuration="1.728997606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:48:09.72711092 +0000 UTC m=+1872.221010745" watchObservedRunningTime="2026-09-11 16:48:09.728997606 +0000 UTC m=+1872.222897418"
Sep 11 16:48:40.718951 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:48:40.718918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:50:23.777773 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.777663    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:50:23.780926 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.780909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.783736 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.783712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 11 16:50:23.784909 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.784890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 11 16:50:23.785039 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.785018    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 11 16:50:23.785125 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.785058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-bcw6h\""
Sep 11 16:50:23.794214 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.794191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:50:23.861250 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.861366 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvfn\" (UniqueName: \"kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.861366 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.861366 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.861473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.861473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.861421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962672 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvfn\" (UniqueName: \"kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.962807 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.962799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.963097 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.963072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.963169 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.963095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.963228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.963175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.965028 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.965009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.965225 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.965209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:23.970616 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:23.970594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvfn\" (UniqueName: \"kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn\") pod \"test-div-v2-kserve-5f6fdb6b77-s6jx5\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:24.091775 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:24.091697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:24.364216 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:24.364191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:50:24.367284 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:50:24.367250    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb53c90_3b39_4ec9_ab1c_76c4b6fbc954.slice/crio-f33466464895aa605c7ea57a959ba3d7cfcd42e015a621d797ebdad0876fcfc6 WatchSource:0}: Error finding container f33466464895aa605c7ea57a959ba3d7cfcd42e015a621d797ebdad0876fcfc6: Status 404 returned error can't find the container with id f33466464895aa605c7ea57a959ba3d7cfcd42e015a621d797ebdad0876fcfc6
Sep 11 16:50:24.369360 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:24.369338    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:50:25.335521 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:25.335486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" event={"ID":"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954","Type":"ContainerStarted","Data":"f33466464895aa605c7ea57a959ba3d7cfcd42e015a621d797ebdad0876fcfc6"}
Sep 11 16:50:25.814149 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:25.814111    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:50:26.343608 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:26.343572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" event={"ID":"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954","Type":"ContainerStarted","Data":"75ac57cbaf9a77bbed9b07d4b91b80db23e4ed1818cf3a5fe7ececf81277901d"}
Sep 11 16:50:26.363818 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:26.363769    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" podStartSLOduration=2.127451492 podStartE2EDuration="3.363754569s" totalImagesPullingTime="1.236303077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:50:26.36205555 +0000 UTC m=+2008.855955386" watchObservedRunningTime="2026-09-11 16:50:26.363754569 +0000 UTC m=+2008.857654381"
Sep 11 16:50:27.092776 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:27.092725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:30.719099 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:30.719061    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:50:34.092524 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:34.092488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:34.104858 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:34.104833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:34.395520 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:34.395427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:50:40.719471 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:40.719428    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:50:40.841126 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:40.841076    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" containerID="cri-o://508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db" gracePeriod=45
Sep 11 16:50:46.358880 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:46.358842    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:50:50.718886 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:50.718841    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:50:50.719288 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:50:50.718981    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:51:00.718877 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:00.718837    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:51:01.379458 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.379396    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerName="main" containerID="cri-o://75ac57cbaf9a77bbed9b07d4b91b80db23e4ed1818cf3a5fe7ececf81277901d" gracePeriod=45
Sep 11 16:51:01.385075 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.385049    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 11 16:51:01.521248 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.521212    2563 generic.go:356] "Generic (PLEG): container finished" podID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerID="75ac57cbaf9a77bbed9b07d4b91b80db23e4ed1818cf3a5fe7ececf81277901d" exitCode=0
Sep 11 16:51:01.521416 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.521290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" event={"ID":"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954","Type":"ContainerDied","Data":"75ac57cbaf9a77bbed9b07d4b91b80db23e4ed1818cf3a5fe7ececf81277901d"}
Sep 11 16:51:01.621924 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.621897    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:51:01.672829 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.672829 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.673009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.673009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.673009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.673009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.672962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvfn\" (UniqueName: \"kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn\") pod \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\" (UID: \"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954\") "
Sep 11 16:51:01.673234 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:01.673234 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:01.673234 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home" (OuterVolumeSpecName: "home") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:01.673354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673240    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:01.673354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673257    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:01.673354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.673274    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:01.675195 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.675172    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn" (OuterVolumeSpecName: "kube-api-access-fhvfn") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "kube-api-access-fhvfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:51:01.675279 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.675179    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm" (OuterVolumeSpecName: "dshm") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:51:01.675279 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.675252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" (UID: "4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:51:01.774530 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.774492    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:01.774530 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.774523    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:01.774530 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:01.774535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvfn\" (UniqueName: \"kubernetes.io/projected/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954-kube-api-access-fhvfn\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:02.527502 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:02.527475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"
Sep 11 16:51:02.527718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:02.527507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5" event={"ID":"4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954","Type":"ContainerDied","Data":"f33466464895aa605c7ea57a959ba3d7cfcd42e015a621d797ebdad0876fcfc6"}
Sep 11 16:51:02.527718 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:02.527561    2563 scope.go:117] "RemoveContainer" containerID="75ac57cbaf9a77bbed9b07d4b91b80db23e4ed1818cf3a5fe7ececf81277901d"
Sep 11 16:51:02.543714 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:02.543668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:51:02.547704 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:02.547673    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-s6jx5"]
Sep 11 16:51:04.042185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:04.042150    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" path="/var/lib/kubelet/pods/4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954/volumes"
Sep 11 16:51:10.718516 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:10.718478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:51:11.084723 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.084692    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:51:11.150850 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.150816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6zg\" (UniqueName: \"kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg\") pod \"d86538e8-4180-4f71-b64c-b9c05a02bffa\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") "
Sep 11 16:51:11.151009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.150882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs\") pod \"d86538e8-4180-4f71-b64c-b9c05a02bffa\" (UID: \"d86538e8-4180-4f71-b64c-b9c05a02bffa\") "
Sep 11 16:51:11.153063 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.153033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d86538e8-4180-4f71-b64c-b9c05a02bffa" (UID: "d86538e8-4180-4f71-b64c-b9c05a02bffa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:51:11.153185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.153044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg" (OuterVolumeSpecName: "kube-api-access-cf6zg") pod "d86538e8-4180-4f71-b64c-b9c05a02bffa" (UID: "d86538e8-4180-4f71-b64c-b9c05a02bffa"). InnerVolumeSpecName "kube-api-access-cf6zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:51:11.251727 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.251658    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86538e8-4180-4f71-b64c-b9c05a02bffa-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:11.251727 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.251720    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6zg\" (UniqueName: \"kubernetes.io/projected/d86538e8-4180-4f71-b64c-b9c05a02bffa-kube-api-access-cf6zg\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:51:11.570223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.570190    2563 generic.go:356] "Generic (PLEG): container finished" podID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerID="508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db" exitCode=0
Sep 11 16:51:11.570223 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.570227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" event={"ID":"d86538e8-4180-4f71-b64c-b9c05a02bffa","Type":"ContainerDied","Data":"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"}
Sep 11 16:51:11.570436 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.570234    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"
Sep 11 16:51:11.570436 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.570269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx" event={"ID":"d86538e8-4180-4f71-b64c-b9c05a02bffa","Type":"ContainerDied","Data":"957e395e4fc0b0a9701d4034c9db2c516b91d8e887e258877177a9d34d85eb7b"}
Sep 11 16:51:11.570436 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.570289    2563 scope.go:117] "RemoveContainer" containerID="508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"
Sep 11 16:51:11.579097 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.579075    2563 scope.go:117] "RemoveContainer" containerID="508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"
Sep 11 16:51:11.579350 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:51:11.579331    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db\": container with ID starting with 508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db not found: ID does not exist" containerID="508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"
Sep 11 16:51:11.579397 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.579361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db"} err="failed to get container status \"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db\": rpc error: code = NotFound desc = could not find container \"508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db\": container with ID starting with 508ec94ccc36da282de0d8fee634401e7d4b9a50c3fe767aaeb25edff53808db not found: ID does not exist"
Sep 11 16:51:11.591760 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.591734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:51:11.597936 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:11.597914    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75xhrnx"]
Sep 11 16:51:12.047026 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:12.046982    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" path="/var/lib/kubelet/pods/d86538e8-4180-4f71-b64c-b9c05a02bffa/volumes"
Sep 11 16:51:24.913500 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913459    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913795    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main"
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main"
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerName="main"
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913829    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerName="main"
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb53c90-3b39-4ec9-ab1c-76c4b6fbc954" containerName="main"
Sep 11 16:51:24.913983 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.913906    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86538e8-4180-4f71-b64c-b9c05a02bffa" containerName="main"
Sep 11 16:51:24.918397 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.918365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:24.922219 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.922192    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-57tm8\""
Sep 11 16:51:24.922352 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.922200    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 11 16:51:24.922352 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.922277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 11 16:51:24.922352 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.922304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 11 16:51:24.922352 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.922279    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-77t59\""
Sep 11 16:51:24.927191 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.927169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:51:24.954390 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.954359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:24.954496 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:24.954481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjdd\" (UniqueName: \"kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.055884 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.055845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.056084 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.055944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjdd\" (UniqueName: \"kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.058381 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.058359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.064535 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.064512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjdd\" (UniqueName: \"kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.229367 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.229330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.499560 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.499534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:51:25.502053 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:51:25.502013    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddcb577_094a_437c_85de_9e4237369850.slice/crio-868a6b0f71eb81c673fe53a2e6a3058ce60d04db8ccfd277d420e32785f1cc3e WatchSource:0}: Error finding container 868a6b0f71eb81c673fe53a2e6a3058ce60d04db8ccfd277d420e32785f1cc3e: Status 404 returned error can't find the container with id 868a6b0f71eb81c673fe53a2e6a3058ce60d04db8ccfd277d420e32785f1cc3e
Sep 11 16:51:25.637418 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.637368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" event={"ID":"dddcb577-094a-437c-85de-9e4237369850","Type":"ContainerStarted","Data":"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"}
Sep 11 16:51:25.637599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.637437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" event={"ID":"dddcb577-094a-437c-85de-9e4237369850","Type":"ContainerStarted","Data":"868a6b0f71eb81c673fe53a2e6a3058ce60d04db8ccfd277d420e32785f1cc3e"}
Sep 11 16:51:25.637599 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.637477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:25.657483 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:25.657432    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podStartSLOduration=1.657416805 podStartE2EDuration="1.657416805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:51:25.655876072 +0000 UTC m=+2068.149775888" watchObservedRunningTime="2026-09-11 16:51:25.657416805 +0000 UTC m=+2068.151316617"
Sep 11 16:51:56.644212 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:56.644174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:51:58.272341 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:58.272311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:51:58.276242 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:51:58.276219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:52:59.394760 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:52:59.394634    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:53:06.644294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:06.644251    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:53:14.414075 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:14.413994    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main" containerID="cri-o://61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52" gracePeriod=45
Sep 11 16:53:16.644241 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:16.644199    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:53:26.644477 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:26.644427    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:53:26.645042 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:26.644546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:53:36.644055 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:36.644013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 11 16:53:44.664455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.664427    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:53:44.756896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.756866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shjdd\" (UniqueName: \"kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd\") pod \"dddcb577-094a-437c-85de-9e4237369850\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") "
Sep 11 16:53:44.757095 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.756947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs\") pod \"dddcb577-094a-437c-85de-9e4237369850\" (UID: \"dddcb577-094a-437c-85de-9e4237369850\") "
Sep 11 16:53:44.759142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.759109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dddcb577-094a-437c-85de-9e4237369850" (UID: "dddcb577-094a-437c-85de-9e4237369850"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:53:44.759142 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.759119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd" (OuterVolumeSpecName: "kube-api-access-shjdd") pod "dddcb577-094a-437c-85de-9e4237369850" (UID: "dddcb577-094a-437c-85de-9e4237369850"). InnerVolumeSpecName "kube-api-access-shjdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:53:44.857817 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.857784    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddcb577-094a-437c-85de-9e4237369850-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:53:44.857817 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:44.857816    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjdd\" (UniqueName: \"kubernetes.io/projected/dddcb577-094a-437c-85de-9e4237369850-kube-api-access-shjdd\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:53:45.295575 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.295545    2563 generic.go:356] "Generic (PLEG): container finished" podID="dddcb577-094a-437c-85de-9e4237369850" containerID="61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52" exitCode=0
Sep 11 16:53:45.295575 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.295581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" event={"ID":"dddcb577-094a-437c-85de-9e4237369850","Type":"ContainerDied","Data":"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"}
Sep 11 16:53:45.295841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.295592    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"
Sep 11 16:53:45.295841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.295623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr" event={"ID":"dddcb577-094a-437c-85de-9e4237369850","Type":"ContainerDied","Data":"868a6b0f71eb81c673fe53a2e6a3058ce60d04db8ccfd277d420e32785f1cc3e"}
Sep 11 16:53:45.295841 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.295644    2563 scope.go:117] "RemoveContainer" containerID="61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"
Sep 11 16:53:45.305020 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.304998    2563 scope.go:117] "RemoveContainer" containerID="61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"
Sep 11 16:53:45.305302 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:53:45.305282    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52\": container with ID starting with 61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52 not found: ID does not exist" containerID="61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"
Sep 11 16:53:45.305353 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.305309    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52"} err="failed to get container status \"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52\": rpc error: code = NotFound desc = could not find container \"61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52\": container with ID starting with 61264ab1af513697e2d1ccb0aea1874fa6311326027f2809623b411429ee9d52 not found: ID does not exist"
Sep 11 16:53:45.316905 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.316881    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:53:45.323558 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:45.323536    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedgb7qr"]
Sep 11 16:53:46.042906 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:53:46.042875    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dddcb577-094a-437c-85de-9e4237369850" path="/var/lib/kubelet/pods/dddcb577-094a-437c-85de-9e4237369850/volumes"
Sep 11 16:54:30.813310 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.813276    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:54:30.813794 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.813641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main"
Sep 11 16:54:30.813794 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.813654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main"
Sep 11 16:54:30.813794 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.813722    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dddcb577-094a-437c-85de-9e4237369850" containerName="main"
Sep 11 16:54:30.816412 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.816396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.819095 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.819073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 11 16:54:30.819207 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.819073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-mzqfb\""
Sep 11 16:54:30.820213 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.820192    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 11 16:54:30.820294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.820195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 11 16:54:30.820294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.820227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 11 16:54:30.820294 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.820195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:54:30.828153 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.828131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:54:30.847255 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847358 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847358 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847358 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847486 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847486 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847486 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.847486 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.847442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbx6\" (UniqueName: \"kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.948646 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.948828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.948828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.948828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.948828 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbx6\" (UniqueName: \"kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.948962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949050 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.949027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949232 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.949082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949295 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.949269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.949333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.949480 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.949460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.951138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.951119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.951314 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.951296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:30.957074 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:30.957055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbx6\" (UniqueName: \"kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6\") pod \"tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:31.029426 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.029396    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:54:31.034574 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.034550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.042874 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.042852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:54:31.049802 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.049780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.049895 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.049822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.049895 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.049861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.049971 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.049903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.127772 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.127671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:31.151053 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151187 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151187 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151292 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151585 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151650 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.151755 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.151737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.153569 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.153549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.348627 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.348584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:54:31.403730 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.403589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:54:31.407454 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:54:31.407427    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e883df_cfd5_4996_90eb_5979dd7dfeff.slice/crio-e06004823b1b8c7be9aec89fbbf87c573767b13507c17e8d00f9701714aa34a1 WatchSource:0}: Error finding container e06004823b1b8c7be9aec89fbbf87c573767b13507c17e8d00f9701714aa34a1: Status 404 returned error can't find the container with id e06004823b1b8c7be9aec89fbbf87c573767b13507c17e8d00f9701714aa34a1
Sep 11 16:54:31.505456 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.505413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerStarted","Data":"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"}
Sep 11 16:54:31.505598 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.505487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerStarted","Data":"e06004823b1b8c7be9aec89fbbf87c573767b13507c17e8d00f9701714aa34a1"}
Sep 11 16:54:31.621624 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:31.620522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:54:31.623660 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:54:31.623620    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f6f90e_4ed6_4091_af96_cf69707404cd.slice/crio-1dc1455fbcaa182b265603bc84933b338b003af1334d219b6123bf5690616524 WatchSource:0}: Error finding container 1dc1455fbcaa182b265603bc84933b338b003af1334d219b6123bf5690616524: Status 404 returned error can't find the container with id 1dc1455fbcaa182b265603bc84933b338b003af1334d219b6123bf5690616524
Sep 11 16:54:32.512367 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:32.512326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerStarted","Data":"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"}
Sep 11 16:54:32.512816 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:32.512394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerStarted","Data":"1dc1455fbcaa182b265603bc84933b338b003af1334d219b6123bf5690616524"}
Sep 11 16:54:34.524949 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:34.524913    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerID="a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250" exitCode=0
Sep 11 16:54:34.525373 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:34.524986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerDied","Data":"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"}
Sep 11 16:54:34.526615 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:34.526591    2563 generic.go:356] "Generic (PLEG): container finished" podID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerID="8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f" exitCode=0
Sep 11 16:54:34.526733 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:34.526661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerDied","Data":"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"}
Sep 11 16:54:35.533663 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:35.533628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerStarted","Data":"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"}
Sep 11 16:54:35.554609 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:35.554560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" podStartSLOduration=3.043871106 podStartE2EDuration="5.554543658s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.510672552s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:54:35.553212183 +0000 UTC m=+2258.047112008" watchObservedRunningTime="2026-09-11 16:54:35.554543658 +0000 UTC m=+2258.048443471"
Sep 11 16:54:36.128228 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:36.128182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:41.127910 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:41.127869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:41.146082 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:41.145752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:54:41.590114 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:54:41.590070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:55:21.819175 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:55:21.819137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerStarted","Data":"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"}
Sep 11 16:55:21.819587 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:55:21.819380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:55:21.837882 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:55:21.837819    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" podStartSLOduration=1.248021579 podStartE2EDuration="50.837767561s" totalImagesPullingTime="47.11116129s" totalInitContainerRuntime="2.478584692s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:55:21.836482616 +0000 UTC m=+2304.330382432" watchObservedRunningTime="2026-09-11 16:55:21.837767561 +0000 UTC m=+2304.331667375"
Sep 11 16:55:52.836165 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:55:52.836080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:56:18.517553 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:18.517523    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2_38f6f90e-4ed6-4091-af96-cf69707404cd/main/0.log"
Sep 11 16:56:40.491549 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.491513    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:56:40.492218 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.491804    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="main" containerID="cri-o://654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6" gracePeriod=30
Sep 11 16:56:40.502466 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.502436    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:56:40.511502 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:40.511472    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:56:40.511502 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.511502 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.511502 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d" cmd=["/bin/sleep","15"]
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.511519    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" message="" err=<
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:40.511532    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main" containerID="cri-o://cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"
Sep 11 16:56:40.511750 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.511567    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main" containerID="cri-o://cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d" gracePeriod=60
Sep 11 16:56:40.574454 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.574409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 11 16:56:40.857852 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:40.857824    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:56:41.006122 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006092    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.006277 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.006277 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glbx6\" (UniqueName: \"kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.006277 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.006446 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006404    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.006527 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.006594 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006546    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.007009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.007009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache\") pod \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\" (UID: \"a4e883df-cfd5-4996-90eb-5979dd7dfeff\") "
Sep 11 16:56:41.007009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.006807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home" (OuterVolumeSpecName: "home") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.007226 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.007226 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:56:41.007339 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007232    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.007339 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007259    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e883df-cfd5-4996-90eb-5979dd7dfeff-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.007339 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007274    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.007339 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.007287    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.009303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.009223    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:56:41.009303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.009256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm" (OuterVolumeSpecName: "dshm") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.011237 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.011211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6" (OuterVolumeSpecName: "kube-api-access-glbx6") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "kube-api-access-glbx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:56:41.035540 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.035461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4e883df-cfd5-4996-90eb-5979dd7dfeff" (UID: "a4e883df-cfd5-4996-90eb-5979dd7dfeff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.061380 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.061355    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:56:41.108066 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.108031    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.108066 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.108061    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glbx6\" (UniqueName: \"kubernetes.io/projected/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kube-api-access-glbx6\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.108229 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.108073    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e883df-cfd5-4996-90eb-5979dd7dfeff-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.108229 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.108083    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4e883df-cfd5-4996-90eb-5979dd7dfeff-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.202490 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.202451    2563 generic.go:356] "Generic (PLEG): container finished" podID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerID="cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d" exitCode=0
Sep 11 16:56:41.202695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.202511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"
Sep 11 16:56:41.202695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.202507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerDied","Data":"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"}
Sep 11 16:56:41.202695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.202571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk" event={"ID":"a4e883df-cfd5-4996-90eb-5979dd7dfeff","Type":"ContainerDied","Data":"e06004823b1b8c7be9aec89fbbf87c573767b13507c17e8d00f9701714aa34a1"}
Sep 11 16:56:41.202695 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.202601    2563 scope.go:117] "RemoveContainer" containerID="cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"
Sep 11 16:56:41.204222 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.204190    2563 generic.go:356] "Generic (PLEG): container finished" podID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerID="654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6" exitCode=0
Sep 11 16:56:41.204323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.204239    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"
Sep 11 16:56:41.204323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.204237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerDied","Data":"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"}
Sep 11 16:56:41.204323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.204305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2" event={"ID":"38f6f90e-4ed6-4091-af96-cf69707404cd","Type":"ContainerDied","Data":"1dc1455fbcaa182b265603bc84933b338b003af1334d219b6123bf5690616524"}
Sep 11 16:56:41.208590 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.208570    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs\") pod \"38f6f90e-4ed6-4091-af96-cf69707404cd\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") "
Sep 11 16:56:41.208722 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.208610    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir\") pod \"38f6f90e-4ed6-4091-af96-cf69707404cd\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") "
Sep 11 16:56:41.208722 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.208632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location\") pod \"38f6f90e-4ed6-4091-af96-cf69707404cd\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") "
Sep 11 16:56:41.208722 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.208652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert\") pod \"38f6f90e-4ed6-4091-af96-cf69707404cd\" (UID: \"38f6f90e-4ed6-4091-af96-cf69707404cd\") "
Sep 11 16:56:41.209045 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.209019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38f6f90e-4ed6-4091-af96-cf69707404cd" (UID: "38f6f90e-4ed6-4091-af96-cf69707404cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:56:41.209155 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.209066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38f6f90e-4ed6-4091-af96-cf69707404cd" (UID: "38f6f90e-4ed6-4091-af96-cf69707404cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.211362 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.211278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38f6f90e-4ed6-4091-af96-cf69707404cd" (UID: "38f6f90e-4ed6-4091-af96-cf69707404cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:56:41.212241 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.212225    2563 scope.go:117] "RemoveContainer" containerID="a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"
Sep 11 16:56:41.219715 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.219670    2563 scope.go:117] "RemoveContainer" containerID="cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"
Sep 11 16:56:41.220030 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:41.220011    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d\": container with ID starting with cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d not found: ID does not exist" containerID="cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"
Sep 11 16:56:41.220087 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.220041    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d"} err="failed to get container status \"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d\": rpc error: code = NotFound desc = could not find container \"cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d\": container with ID starting with cf75c94266ad7ff02117f5175439bd2a35828a12c8216888eb4de5a41321554d not found: ID does not exist"
Sep 11 16:56:41.220087 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.220063    2563 scope.go:117] "RemoveContainer" containerID="a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"
Sep 11 16:56:41.220338 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:41.220320    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250\": container with ID starting with a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250 not found: ID does not exist" containerID="a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"
Sep 11 16:56:41.220405 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.220346    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250"} err="failed to get container status \"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250\": rpc error: code = NotFound desc = could not find container \"a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250\": container with ID starting with a56be5681121b08e0c92a459da91ecbed7e491a95372143d0ff37333e26fc250 not found: ID does not exist"
Sep 11 16:56:41.220405 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.220366    2563 scope.go:117] "RemoveContainer" containerID="654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"
Sep 11 16:56:41.225359 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.225338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:56:41.228764 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.228744    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bc5bdc78c-vz9rk"]
Sep 11 16:56:41.230250 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.230227    2563 scope.go:117] "RemoveContainer" containerID="8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"
Sep 11 16:56:41.234240 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.234213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38f6f90e-4ed6-4091-af96-cf69707404cd" (UID: "38f6f90e-4ed6-4091-af96-cf69707404cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:56:41.237749 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.237733    2563 scope.go:117] "RemoveContainer" containerID="654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"
Sep 11 16:56:41.238013 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:41.237996    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6\": container with ID starting with 654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6 not found: ID does not exist" containerID="654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"
Sep 11 16:56:41.238058 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.238019    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6"} err="failed to get container status \"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6\": rpc error: code = NotFound desc = could not find container \"654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6\": container with ID starting with 654a54f805b5edf4a066ace215f9cbe927bc3713dd599f0b1520505bbd83d0c6 not found: ID does not exist"
Sep 11 16:56:41.238058 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.238035    2563 scope.go:117] "RemoveContainer" containerID="8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"
Sep 11 16:56:41.238297 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:56:41.238280    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f\": container with ID starting with 8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f not found: ID does not exist" containerID="8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"
Sep 11 16:56:41.238341 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.238302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f"} err="failed to get container status \"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f\": rpc error: code = NotFound desc = could not find container \"8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f\": container with ID starting with 8e71009cd0b814377972a731bcc5a3f3cc0322a0ce7b783aad90afef45cc6e3f not found: ID does not exist"
Sep 11 16:56:41.309435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.309372    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f6f90e-4ed6-4091-af96-cf69707404cd-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.309435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.309395    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.309435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.309405    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f6f90e-4ed6-4091-af96-cf69707404cd-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.309435 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.309415    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f6f90e-4ed6-4091-af96-cf69707404cd-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:56:41.527474 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.527442    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:56:41.529993 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:41.529970    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-66c96f946b-rwsv2"]
Sep 11 16:56:42.042571 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:42.042536    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" path="/var/lib/kubelet/pods/38f6f90e-4ed6-4091-af96-cf69707404cd/volumes"
Sep 11 16:56:42.043023 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:42.043009    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" path="/var/lib/kubelet/pods/a4e883df-cfd5-4996-90eb-5979dd7dfeff/volumes"
Sep 11 16:56:58.300498 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:58.300469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:56:58.305797 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:56:58.305771    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 16:57:59.310691 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.310654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="storage-initializer"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311032    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="storage-initializer"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311044    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="storage-initializer"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311050    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="storage-initializer"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="main"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311068    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="main"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311080    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main"
Sep 11 16:57:59.311103 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311085    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main"
Sep 11 16:57:59.311354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311141    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e883df-cfd5-4996-90eb-5979dd7dfeff" containerName="main"
Sep 11 16:57:59.311354 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.311150    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f6f90e-4ed6-4091-af96-cf69707404cd" containerName="main"
Sep 11 16:57:59.314264 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.314242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.317323 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.317301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 11 16:57:59.317427 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.317309    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:57:59.317473 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.317309    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 11 16:57:59.318654 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.318573    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 11 16:57:59.318654 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.318594    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-bqx9c\""
Sep 11 16:57:59.318654 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.318605    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 11 16:57:59.325832 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.325810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:57:59.373391 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373519 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373610 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6829x\" (UniqueName: \"kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373774 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.373774 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.373671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474513 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474513 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6829x\" (UniqueName: \"kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.474731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475009 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.474976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475124 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.475086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475180 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.475152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.475158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.475373 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.475353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.476932 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.476907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.477190 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.477171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.484957 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.484935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6829x\" (UniqueName: \"kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x\") pod \"tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.608953 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.608870    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:57:59.613612 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.613588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.625120 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.625082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:57:59.625265 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.625196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:57:59.676277 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.676244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.676396 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.676362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.676466 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.676396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.676525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.676459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.777964 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.777931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.777978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.777999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778139 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.778019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778359 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.778336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778359 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.778353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.778732 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.778711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.780370 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.780353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:57:59.902107 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.902082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:57:59.903889 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:57:59.903861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb943303_3d6c_493b_85e4_4ad587476cb0.slice/crio-771040c3ec6b18b9b2a1a0ebb86f6333b84c9205d68bca5c2bd6a9aa65e34f8a WatchSource:0}: Error finding container 771040c3ec6b18b9b2a1a0ebb86f6333b84c9205d68bca5c2bd6a9aa65e34f8a: Status 404 returned error can't find the container with id 771040c3ec6b18b9b2a1a0ebb86f6333b84c9205d68bca5c2bd6a9aa65e34f8a
Sep 11 16:57:59.926096 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:57:59.926075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:58:00.199397 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:00.199362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:58:00.202357 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:58:00.202333    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f80964_16f1_4c70_b694_1de7ad10082b.slice/crio-5944d1f0e4d6a6027f7e85eb839eee5eacd2bd947adcf3cb11b266b6f0550ff1 WatchSource:0}: Error finding container 5944d1f0e4d6a6027f7e85eb839eee5eacd2bd947adcf3cb11b266b6f0550ff1: Status 404 returned error can't find the container with id 5944d1f0e4d6a6027f7e85eb839eee5eacd2bd947adcf3cb11b266b6f0550ff1
Sep 11 16:58:00.583217 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:00.583119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerStarted","Data":"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"}
Sep 11 16:58:00.583217 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:00.583192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerStarted","Data":"5944d1f0e4d6a6027f7e85eb839eee5eacd2bd947adcf3cb11b266b6f0550ff1"}
Sep 11 16:58:00.585303 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:00.585274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerStarted","Data":"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"}
Sep 11 16:58:00.585425 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:00.585324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerStarted","Data":"771040c3ec6b18b9b2a1a0ebb86f6333b84c9205d68bca5c2bd6a9aa65e34f8a"}
Sep 11 16:58:02.596986 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:02.596948    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerID="dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070" exitCode=0
Sep 11 16:58:02.597412 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:02.597010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerDied","Data":"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"}
Sep 11 16:58:03.605016 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:03.604977    2563 generic.go:356] "Generic (PLEG): container finished" podID="99f80964-16f1-4c70-b694-1de7ad10082b" containerID="d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992" exitCode=0
Sep 11 16:58:03.605456 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:03.605053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerDied","Data":"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"}
Sep 11 16:58:03.607000 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:03.606978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerStarted","Data":"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"}
Sep 11 16:58:03.625406 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:03.625381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:03.639602 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:03.639532    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" podStartSLOduration=2.143184593 podStartE2EDuration="4.639515513s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.49633092s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:03.639326883 +0000 UTC m=+2466.133226721" watchObservedRunningTime="2026-09-11 16:58:03.639515513 +0000 UTC m=+2466.133415327"
Sep 11 16:58:04.614174 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:04.614132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerStarted","Data":"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"}
Sep 11 16:58:04.614670 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:04.614495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:58:04.635081 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:04.635027    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" podStartSLOduration=3.172152947 podStartE2EDuration="5.635012694s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.462859747s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:58:04.633869148 +0000 UTC m=+2467.127768973" watchObservedRunningTime="2026-09-11 16:58:04.635012694 +0000 UTC m=+2467.128912506"
Sep 11 16:58:09.625723 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:09.625670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:09.638552 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:09.638526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:09.663649 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:09.663621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:35.633923 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:35.633890    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:58:39.412381 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:39.412326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t_99f80964-16f1-4c70-b694-1de7ad10082b/main/0.log"
Sep 11 16:58:42.816313 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:42.816281    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:58:42.821790 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:42.821732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:58:42.822086 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:42.821995    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="main" containerID="cri-o://8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce" gracePeriod=30
Sep 11 16:58:42.827280 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:42.827241    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 16:58:42.827280 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.827280 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.827280 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2" cmd=["/bin/sleep","15"]
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:42.827289    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" message="" err=<
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:42.827302    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="main" containerID="cri-o://7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"
Sep 11 16:58:42.827472 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:42.827332    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="main" containerID="cri-o://7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2" gracePeriod=60
Sep 11 16:58:43.169878 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.169852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:43.253990 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.253954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.253997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254131    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254177 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6829x\" (UniqueName: \"kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location\") pod \"bb943303-3d6c-493b-85e4-4ad587476cb0\" (UID: \"bb943303-3d6c-493b-85e4-4ad587476cb0\") "
Sep 11 16:58:43.254503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home" (OuterVolumeSpecName: "home") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.254503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.254503 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.254809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254509    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.254809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254528    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.254809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254544    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.254809 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.254480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:58:43.256777 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.256750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm" (OuterVolumeSpecName: "dshm") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.256879 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.256786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:43.256879 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.256791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x" (OuterVolumeSpecName: "kube-api-access-6829x") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "kube-api-access-6829x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:58:43.277622 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.277590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb943303-3d6c-493b-85e4-4ad587476cb0" (UID: "bb943303-3d6c-493b-85e4-4ad587476cb0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.355138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.355073    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.355138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.355100    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb943303-3d6c-493b-85e4-4ad587476cb0-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.355138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.355112    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb943303-3d6c-493b-85e4-4ad587476cb0-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.355138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.355121    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb943303-3d6c-493b-85e4-4ad587476cb0-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.355138 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.355129    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6829x\" (UniqueName: \"kubernetes.io/projected/bb943303-3d6c-493b-85e4-4ad587476cb0-kube-api-access-6829x\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.462650 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.462625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:58:43.555922 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.555889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert\") pod \"99f80964-16f1-4c70-b694-1de7ad10082b\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") "
Sep 11 16:58:43.555922 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.555923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs\") pod \"99f80964-16f1-4c70-b694-1de7ad10082b\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") "
Sep 11 16:58:43.556162 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.555981    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location\") pod \"99f80964-16f1-4c70-b694-1de7ad10082b\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") "
Sep 11 16:58:43.556162 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.556046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir\") pod \"99f80964-16f1-4c70-b694-1de7ad10082b\" (UID: \"99f80964-16f1-4c70-b694-1de7ad10082b\") "
Sep 11 16:58:43.556279 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.556202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99f80964-16f1-4c70-b694-1de7ad10082b" (UID: "99f80964-16f1-4c70-b694-1de7ad10082b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:58:43.556345 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.556313    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f80964-16f1-4c70-b694-1de7ad10082b-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.556460 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.556438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99f80964-16f1-4c70-b694-1de7ad10082b" (UID: "99f80964-16f1-4c70-b694-1de7ad10082b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.558176 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.558156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99f80964-16f1-4c70-b694-1de7ad10082b" (UID: "99f80964-16f1-4c70-b694-1de7ad10082b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:58:43.577977 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.577948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99f80964-16f1-4c70-b694-1de7ad10082b" (UID: "99f80964-16f1-4c70-b694-1de7ad10082b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:58:43.657554 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.657481    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.657554 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.657505    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f80964-16f1-4c70-b694-1de7ad10082b-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.657554 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.657517    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f80964-16f1-4c70-b694-1de7ad10082b-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 16:58:43.819309 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.819270    2563 generic.go:356] "Generic (PLEG): container finished" podID="99f80964-16f1-4c70-b694-1de7ad10082b" containerID="8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce" exitCode=0
Sep 11 16:58:43.819808 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.819349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerDied","Data":"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"}
Sep 11 16:58:43.819808 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.819416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t" event={"ID":"99f80964-16f1-4c70-b694-1de7ad10082b","Type":"ContainerDied","Data":"5944d1f0e4d6a6027f7e85eb839eee5eacd2bd947adcf3cb11b266b6f0550ff1"}
Sep 11 16:58:43.819808 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.819448    2563 scope.go:117] "RemoveContainer" containerID="8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"
Sep 11 16:58:43.819808 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.819473    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"
Sep 11 16:58:43.821410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.821381    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerID="7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2" exitCode=0
Sep 11 16:58:43.821525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.821450    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"
Sep 11 16:58:43.821525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.821459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerDied","Data":"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"}
Sep 11 16:58:43.821525 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.821502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs" event={"ID":"bb943303-3d6c-493b-85e4-4ad587476cb0","Type":"ContainerDied","Data":"771040c3ec6b18b9b2a1a0ebb86f6333b84c9205d68bca5c2bd6a9aa65e34f8a"}
Sep 11 16:58:43.830471 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.830303    2563 scope.go:117] "RemoveContainer" containerID="d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"
Sep 11 16:58:43.838565 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.838545    2563 scope.go:117] "RemoveContainer" containerID="8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"
Sep 11 16:58:43.838848 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:43.838828    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce\": container with ID starting with 8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce not found: ID does not exist" containerID="8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"
Sep 11 16:58:43.838903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.838857    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce"} err="failed to get container status \"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce\": rpc error: code = NotFound desc = could not find container \"8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce\": container with ID starting with 8f6988f13cc3e62cd12b1d50c70c9cf64dc40999b7c75c9f9950328ec40ff2ce not found: ID does not exist"
Sep 11 16:58:43.838903 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.838872    2563 scope.go:117] "RemoveContainer" containerID="d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"
Sep 11 16:58:43.839141 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:43.839123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992\": container with ID starting with d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992 not found: ID does not exist" containerID="d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"
Sep 11 16:58:43.839208 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.839150    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992"} err="failed to get container status \"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992\": rpc error: code = NotFound desc = could not find container \"d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992\": container with ID starting with d960c5d5e1ef3bd12d938d53f08c0edb94323f4096c5fcdbf3b9c9dfb7377992 not found: ID does not exist"
Sep 11 16:58:43.839208 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.839173    2563 scope.go:117] "RemoveContainer" containerID="7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"
Sep 11 16:58:43.843031 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.843009    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:58:43.847263 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.847244    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7d59975f64-9z77t"]
Sep 11 16:58:43.847847 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.847830    2563 scope.go:117] "RemoveContainer" containerID="dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"
Sep 11 16:58:43.855299 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.855280    2563 scope.go:117] "RemoveContainer" containerID="7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"
Sep 11 16:58:43.855528 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:43.855510    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2\": container with ID starting with 7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2 not found: ID does not exist" containerID="7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"
Sep 11 16:58:43.855580 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.855535    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2"} err="failed to get container status \"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2\": rpc error: code = NotFound desc = could not find container \"7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2\": container with ID starting with 7fcc846d93a2f9eccd61a09f5a2c7d8e1ba5c18eb6dd6d4b0164c169824db1c2 not found: ID does not exist"
Sep 11 16:58:43.855580 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.855557    2563 scope.go:117] "RemoveContainer" containerID="dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"
Sep 11 16:58:43.855826 ip-10-0-128-144 kubenswrapper[2563]: E0911 16:58:43.855808    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070\": container with ID starting with dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070 not found: ID does not exist" containerID="dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"
Sep 11 16:58:43.855878 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.855833    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070"} err="failed to get container status \"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070\": rpc error: code = NotFound desc = could not find container \"dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070\": container with ID starting with dc0aa32cff3b56f9b7e8a732ee736b10bc3320ce740a39c64cde5ec0b349b070 not found: ID does not exist"
Sep 11 16:58:43.858769 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.858747    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:58:43.864176 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:43.864155    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9459bfdfd-c6hqs"]
Sep 11 16:58:44.041972 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:44.041939    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" path="/var/lib/kubelet/pods/99f80964-16f1-4c70-b694-1de7ad10082b/volumes"
Sep 11 16:58:44.042433 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:58:44.042414    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" path="/var/lib/kubelet/pods/bb943303-3d6c-493b-85e4-4ad587476cb0/volumes"
Sep 11 16:59:36.412667 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.412634    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.412977    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="storage-initializer"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.412990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="storage-initializer"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.412999    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="storage-initializer"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413004    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="storage-initializer"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="main"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413029    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="main"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413038    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="main"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="main"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413095    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb943303-3d6c-493b-85e4-4ad587476cb0" containerName="main"
Sep 11 16:59:36.413185 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.413106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f80964-16f1-4c70-b694-1de7ad10082b" containerName="main"
Sep 11 16:59:36.416273 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.416253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.418955 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.418925    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 11 16:59:36.419079 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.418930    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 16:59:36.419079 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.418974    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 11 16:59:36.419079 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.418989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-jcvlw\""
Sep 11 16:59:36.420171 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.420151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 11 16:59:36.420344 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.420326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 11 16:59:36.428199 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.428175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 16:59:36.498395 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdtl\" (UniqueName: \"kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498545 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498744 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.498744 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.498606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.599897 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.599861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.599914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.599945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.599983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600065 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600344 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600344 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdtl\" (UniqueName: \"kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600344 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600514 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600824 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.600935 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.600876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.601160 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.601139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.603006 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.602981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.603369 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.603334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.612306 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.612284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdtl\" (UniqueName: \"kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl\") pod \"tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.617056 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.617036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 16:59:36.621635 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.621615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.634382 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.634141    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 16:59:36.700910 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.700880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.701069 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.700929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.701069 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.701007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.701157 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.701067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.726934 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.726900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:36.803052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.802809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.802869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.802916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803052 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.802962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803455 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.803428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803573 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.803549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.803853 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.803833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.805644 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.805623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65555bb476-z87fw\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:36.936581 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:36.936548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:37.005360 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:37.005326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 16:59:37.006825 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:59:37.006791    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a0915e_9cb6_4e1d_8974_e87d1c3fe48c.slice/crio-94ac04592c81293eec2842491bc038316d5d202010e62323853f7e4dc8da12df WatchSource:0}: Error finding container 94ac04592c81293eec2842491bc038316d5d202010e62323853f7e4dc8da12df: Status 404 returned error can't find the container with id 94ac04592c81293eec2842491bc038316d5d202010e62323853f7e4dc8da12df
Sep 11 16:59:37.064295 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:37.064262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerStarted","Data":"94ac04592c81293eec2842491bc038316d5d202010e62323853f7e4dc8da12df"}
Sep 11 16:59:37.208206 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:37.208177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 16:59:37.210488 ip-10-0-128-144 kubenswrapper[2563]: W0911 16:59:37.210462    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb883288a_ebbf_467d_9f44_735fb9859f58.slice/crio-8ad216ab252cd993e48de1cfdcdc39c6de73f6a1b65cd7f7826f529412bdb3e3 WatchSource:0}: Error finding container 8ad216ab252cd993e48de1cfdcdc39c6de73f6a1b65cd7f7826f529412bdb3e3: Status 404 returned error can't find the container with id 8ad216ab252cd993e48de1cfdcdc39c6de73f6a1b65cd7f7826f529412bdb3e3
Sep 11 16:59:38.070420 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:38.070378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerStarted","Data":"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"}
Sep 11 16:59:38.070912 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:38.070438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerStarted","Data":"8ad216ab252cd993e48de1cfdcdc39c6de73f6a1b65cd7f7826f529412bdb3e3"}
Sep 11 16:59:38.072150 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:38.072126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerStarted","Data":"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"}
Sep 11 16:59:40.083795 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:40.083755    2563 generic.go:356] "Generic (PLEG): container finished" podID="b883288a-ebbf-467d-9f44-735fb9859f58" containerID="e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e" exitCode=0
Sep 11 16:59:40.084274 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:40.083837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerDied","Data":"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"}
Sep 11 16:59:40.085419 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:40.085397    2563 generic.go:356] "Generic (PLEG): container finished" podID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerID="7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f" exitCode=0
Sep 11 16:59:40.085513 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:40.085467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerDied","Data":"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"}
Sep 11 16:59:41.093731 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.093668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerStarted","Data":"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"}
Sep 11 16:59:41.094231 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.093908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 16:59:41.095643 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.095615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerStarted","Data":"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"}
Sep 11 16:59:41.111896 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.111851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" podStartSLOduration=2.625803998 podStartE2EDuration="5.111835543s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.486031545s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:41.110628581 +0000 UTC m=+2563.604528395" watchObservedRunningTime="2026-09-11 16:59:41.111835543 +0000 UTC m=+2563.605735354"
Sep 11 16:59:41.128614 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.128566    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" podStartSLOduration=2.61831177 podStartE2EDuration="5.128526046s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.510214276s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:59:41.128195938 +0000 UTC m=+2563.622095751" watchObservedRunningTime="2026-09-11 16:59:41.128526046 +0000 UTC m=+2563.622425892"
Sep 11 16:59:41.727268 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:41.727231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:46.727215 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:46.727171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:46.745410 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:46.745378    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 16:59:47.136889 ip-10-0-128-144 kubenswrapper[2563]: I0911 16:59:47.136811    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 17:00:12.112066 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:12.112031    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 17:00:27.803611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:27.803575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-65555bb476-z87fw_b883288a-ebbf-467d-9f44-735fb9859f58/main/0.log"
Sep 11 17:00:58.407674 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:58.407638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 17:00:58.412755 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:58.412727    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 17:00:58.412962 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:58.412926    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="main" containerID="cri-o://046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc" gracePeriod=30
Sep 11 17:00:58.417076 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:58.417045    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:00:58.417076 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.417076 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.417076 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3" cmd=["/bin/sleep","15"]
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:58.417093    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" message="" err=<
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:58.417106    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main" containerID="cri-o://49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"
Sep 11 17:00:58.417302 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:58.417146    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main" containerID="cri-o://49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3" gracePeriod=60
Sep 11 17:00:59.001307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.001276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 17:00:59.133264 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs\") pod \"b883288a-ebbf-467d-9f44-735fb9859f58\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") "
Sep 11 17:00:59.133418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert\") pod \"b883288a-ebbf-467d-9f44-735fb9859f58\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") "
Sep 11 17:00:59.133418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location\") pod \"b883288a-ebbf-467d-9f44-735fb9859f58\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") "
Sep 11 17:00:59.133418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir\") pod \"b883288a-ebbf-467d-9f44-735fb9859f58\" (UID: \"b883288a-ebbf-467d-9f44-735fb9859f58\") "
Sep 11 17:00:59.133732 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b883288a-ebbf-467d-9f44-735fb9859f58" (UID: "b883288a-ebbf-467d-9f44-735fb9859f58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:59.133847 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.133826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b883288a-ebbf-467d-9f44-735fb9859f58" (UID: "b883288a-ebbf-467d-9f44-735fb9859f58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.135569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.135541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b883288a-ebbf-467d-9f44-735fb9859f58" (UID: "b883288a-ebbf-467d-9f44-735fb9859f58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:59.152421 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.152401    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 17:00:59.157846 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.157822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b883288a-ebbf-467d-9f44-735fb9859f58" (UID: "b883288a-ebbf-467d-9f44-735fb9859f58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.234590 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.234795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.234795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.234795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqdtl\" (UniqueName: \"kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.234795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.234795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.235087 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.235087 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache\") pod \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\" (UID: \"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c\") "
Sep 11 17:00:59.235087 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.234986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:00:59.235234 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home" (OuterVolumeSpecName: "home") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.235234 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235203    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235234 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235224    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b883288a-ebbf-467d-9f44-735fb9859f58-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235240    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache" (OuterVolumeSpecName: "model-cache") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235255    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235294    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b883288a-ebbf-467d-9f44-735fb9859f58-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235312    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b883288a-ebbf-467d-9f44-735fb9859f58-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.235394 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.235266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.236840 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.236811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm" (OuterVolumeSpecName: "dshm") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.236939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.236866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:00:59.236939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.236923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl" (OuterVolumeSpecName: "kube-api-access-fqdtl") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "kube-api-access-fqdtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:00:59.256735 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.256708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" (UID: "22a0915e-9cb6-4e1d-8974-e87d1c3fe48c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336579    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336606    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336616    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdtl\" (UniqueName: \"kubernetes.io/projected/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kube-api-access-fqdtl\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336625    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336634    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.336642 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.336643    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:00:59.474524 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.474490    2563 generic.go:356] "Generic (PLEG): container finished" podID="b883288a-ebbf-467d-9f44-735fb9859f58" containerID="046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc" exitCode=0
Sep 11 17:00:59.474918 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.474543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"
Sep 11 17:00:59.474918 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.474572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerDied","Data":"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"}
Sep 11 17:00:59.474918 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.474635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw" event={"ID":"b883288a-ebbf-467d-9f44-735fb9859f58","Type":"ContainerDied","Data":"8ad216ab252cd993e48de1cfdcdc39c6de73f6a1b65cd7f7826f529412bdb3e3"}
Sep 11 17:00:59.474918 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.474669    2563 scope.go:117] "RemoveContainer" containerID="046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"
Sep 11 17:00:59.476373 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.476347    2563 generic.go:356] "Generic (PLEG): container finished" podID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerID="49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3" exitCode=0
Sep 11 17:00:59.476474 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.476398    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"
Sep 11 17:00:59.476474 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.476405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerDied","Data":"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"}
Sep 11 17:00:59.476474 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.476454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" event={"ID":"22a0915e-9cb6-4e1d-8974-e87d1c3fe48c","Type":"ContainerDied","Data":"94ac04592c81293eec2842491bc038316d5d202010e62323853f7e4dc8da12df"}
Sep 11 17:00:59.485605 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.485583    2563 scope.go:117] "RemoveContainer" containerID="e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"
Sep 11 17:00:59.493750 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.493732    2563 scope.go:117] "RemoveContainer" containerID="046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"
Sep 11 17:00:59.493990 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:59.493972    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc\": container with ID starting with 046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc not found: ID does not exist" containerID="046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"
Sep 11 17:00:59.494040 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.493998    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc"} err="failed to get container status \"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc\": rpc error: code = NotFound desc = could not find container \"046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc\": container with ID starting with 046aa9831f43e237350db2beb72cc01cd6b17b373586a3eae4c39d5d0b765efc not found: ID does not exist"
Sep 11 17:00:59.494040 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.494016    2563 scope.go:117] "RemoveContainer" containerID="e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"
Sep 11 17:00:59.494253 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:59.494233    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e\": container with ID starting with e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e not found: ID does not exist" containerID="e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"
Sep 11 17:00:59.494299 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.494259    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e"} err="failed to get container status \"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e\": rpc error: code = NotFound desc = could not find container \"e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e\": container with ID starting with e47cc15cb14507ce6b5deffeeb5ffad0980f120ff847bdcd777d5b823db5734e not found: ID does not exist"
Sep 11 17:00:59.494299 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.494277    2563 scope.go:117] "RemoveContainer" containerID="49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"
Sep 11 17:00:59.499045 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.499026    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 17:00:59.503002 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.502979    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65555bb476-z87fw"]
Sep 11 17:00:59.503383 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.503371    2563 scope.go:117] "RemoveContainer" containerID="7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"
Sep 11 17:00:59.510984 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.510969    2563 scope.go:117] "RemoveContainer" containerID="49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"
Sep 11 17:00:59.511228 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:59.511212    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3\": container with ID starting with 49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3 not found: ID does not exist" containerID="49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"
Sep 11 17:00:59.511272 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.511235    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3"} err="failed to get container status \"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3\": rpc error: code = NotFound desc = could not find container \"49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3\": container with ID starting with 49f970c8bac238e6a3f0adf0ef928541279726800279657388f77eb7e24781c3 not found: ID does not exist"
Sep 11 17:00:59.511272 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.511250    2563 scope.go:117] "RemoveContainer" containerID="7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"
Sep 11 17:00:59.511503 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:00:59.511485    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f\": container with ID starting with 7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f not found: ID does not exist" containerID="7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"
Sep 11 17:00:59.511548 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.511507    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f"} err="failed to get container status \"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f\": rpc error: code = NotFound desc = could not find container \"7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f\": container with ID starting with 7c2a8b2c655135c32a3bc9ff95d5828e14c08032b94f7b26aae6c6095558bf9f not found: ID does not exist"
Sep 11 17:00:59.514795 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.514772    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 17:00:59.521181 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:00:59.521159    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh"]
Sep 11 17:01:00.046271 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:01:00.046236    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" path="/var/lib/kubelet/pods/22a0915e-9cb6-4e1d-8974-e87d1c3fe48c/volumes"
Sep 11 17:01:00.046762 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:01:00.046741    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" path="/var/lib/kubelet/pods/b883288a-ebbf-467d-9f44-735fb9859f58/volumes"
Sep 11 17:01:00.126276 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:01:00.126237    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5bbc8d4f9d-kr5rh" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 17:01:58.329516 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:01:58.329490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:01:58.339753 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:01:58.339734    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:04:45.597471 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597435    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597789    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="main"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="main"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="storage-initializer"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597830    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="storage-initializer"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597844    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597856    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="storage-initializer"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597861    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="storage-initializer"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b883288a-ebbf-467d-9f44-735fb9859f58" containerName="main"
Sep 11 17:04:45.597980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.597924    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a0915e-9cb6-4e1d-8974-e87d1c3fe48c" containerName="main"
Sep 11 17:04:45.600886 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.600867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.603981 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.603957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 11 17:04:45.604111 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.604001    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:04:45.604111 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.604009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-hxcfg\""
Sep 11 17:04:45.605147 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.605125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 11 17:04:45.615524 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.615500    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:04:45.714561 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714763 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714763 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714763 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714763 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714923 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714923 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5dv\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714923 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.714923 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.714857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815539 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5dv\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.815751 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.815820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.816003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.816054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816272 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.816176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816272 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.816199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.816526 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.816506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.818471 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.818444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.818595 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.818579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.823546 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.823524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.823924 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.823902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5dv\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv\") pod \"router-gateway-1-openshift-default-55f8b478f9-lgp7r\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:45.912832 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:45.912758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:04:46.191208 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:46.191183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:04:46.193441 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:04:46.193415    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf326325a_b36f_465b_a518_bf121ea21b1f.slice/crio-1e8c92c8b01203d3f7d0a5df4600f47ff31cd84e2838dddbd1d8cc5979f23bac WatchSource:0}: Error finding container 1e8c92c8b01203d3f7d0a5df4600f47ff31cd84e2838dddbd1d8cc5979f23bac: Status 404 returned error can't find the container with id 1e8c92c8b01203d3f7d0a5df4600f47ff31cd84e2838dddbd1d8cc5979f23bac
Sep 11 17:04:46.195595 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:46.195577    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:04:46.529401 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:46.529325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerStarted","Data":"1e8c92c8b01203d3f7d0a5df4600f47ff31cd84e2838dddbd1d8cc5979f23bac"}
Sep 11 17:04:57.474335 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.474254    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:04:57.477690 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.477663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.480120 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.480096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 11 17:04:57.480260 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.480240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 11 17:04:57.481299 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.481275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-ptrfr\""
Sep 11 17:04:57.481401 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.481314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 11 17:04:57.485706 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.485665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:04:57.617697 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.617892 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.617892 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.617892 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn69c\" (UniqueName: \"kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.618062 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.618062 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.617952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718785 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718986 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718986 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718986 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718986 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.718986 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.718980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn69c\" (UniqueName: \"kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.719247 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.719223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.719489 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.719461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.719603 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.719584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.721963 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.721938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.722090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.721940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.728035 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.727971    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn69c\" (UniqueName: \"kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:57.792548 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:57.792516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:58.078988 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:58.078926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:04:58.081435 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:04:58.081406    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7716dc96_2a29_410f_82ae_72d0d82c0175.slice/crio-01dec0fc7341439fdebfcd8ca2b364374d415de6f2f58c023e30fdafb2228d07 WatchSource:0}: Error finding container 01dec0fc7341439fdebfcd8ca2b364374d415de6f2f58c023e30fdafb2228d07: Status 404 returned error can't find the container with id 01dec0fc7341439fdebfcd8ca2b364374d415de6f2f58c023e30fdafb2228d07
Sep 11 17:04:58.587495 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:58.587439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" event={"ID":"7716dc96-2a29-410f-82ae-72d0d82c0175","Type":"ContainerStarted","Data":"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"}
Sep 11 17:04:58.588010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:58.587519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" event={"ID":"7716dc96-2a29-410f-82ae-72d0d82c0175","Type":"ContainerStarted","Data":"01dec0fc7341439fdebfcd8ca2b364374d415de6f2f58c023e30fdafb2228d07"}
Sep 11 17:04:58.607940 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:58.607892    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" podStartSLOduration=1.6078766 podStartE2EDuration="1.6078766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:58.60566226 +0000 UTC m=+2881.099562073" watchObservedRunningTime="2026-09-11 17:04:58.6078766 +0000 UTC m=+2881.101776412"
Sep 11 17:04:58.793505 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:58.793447    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:04:59.279839 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.279798    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:04:59.279947 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.279878    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:04:59.279947 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.279908    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:04:59.596697 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.596601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerStarted","Data":"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"}
Sep 11 17:04:59.623247 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.623180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podStartSLOduration=1.53933615 podStartE2EDuration="14.62315825s" totalImagesPullingTime="13.0838221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:04:59.620156428 +0000 UTC m=+2882.114056243" watchObservedRunningTime="2026-09-11 17:04:59.62315825 +0000 UTC m=+2882.117058063"
Sep 11 17:04:59.913693 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:04:59.913592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:00.913792 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:00.913757    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:00.919587 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:00.919555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:01.608119 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:01.608087    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:07.793030 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:07.792996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:05:07.805043 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:07.805019    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:05:08.658894 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:08.658860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:05:32.649822 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:32.649771    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 17:05:34.782038 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.781998    2563 generic.go:356] "Generic (PLEG): container finished" podID="f326325a-b36f-465b-a518-bf121ea21b1f" containerID="6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971" exitCode=0
Sep 11 17:05:34.782472 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.782066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerDied","Data":"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"}
Sep 11 17:05:34.782472 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.782460    2563 scope.go:117] "RemoveContainer" containerID="6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"
Sep 11 17:05:34.783405 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.783375    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:05:34.783462 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.783448    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:05:34.783499 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:34.783482    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:05:35.788366 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:35.788331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerStarted","Data":"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"}
Sep 11 17:05:35.913828 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:35.913790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:35.914006 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:35.913977    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:36.913221 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:36.913180    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:37.913997 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:37.913954    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:38.913094 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:38.913048    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:39.913324 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:39.913271    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:40.913931 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:40.913884    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 17:05:41.917058 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:41.917027    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:41.917446 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:41.917391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:41.918168 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:41.918142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:05:50.912784 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:50.912741    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:05:50.924603 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:05:50.924556    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:05:50.924603 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:50.924603 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:50.924603 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35" cmd=["/bin/sleep","15"]
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:50.924626    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" message="" err=<
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:05:50.924643    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerName="main" containerID="cri-o://947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"
Sep 11 17:05:50.924926 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:50.924709    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerName="main" containerID="cri-o://947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35" gracePeriod=60
Sep 11 17:05:51.275136 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.275109    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:05:51.382201 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn69c\" (UniqueName: \"kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382396 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382396 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382302    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382396 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382334    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382396 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382396 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm\") pod \"7716dc96-2a29-410f-82ae-72d0d82c0175\" (UID: \"7716dc96-2a29-410f-82ae-72d0d82c0175\") "
Sep 11 17:05:51.382653 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382603    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:51.382653 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache" (OuterVolumeSpecName: "model-cache") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:51.382653 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.382610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home" (OuterVolumeSpecName: "home") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:51.384586 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.384556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm" (OuterVolumeSpecName: "dshm") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:05:51.384741 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.384616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:05:51.384741 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.384629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c" (OuterVolumeSpecName: "kube-api-access-hn69c") pod "7716dc96-2a29-410f-82ae-72d0d82c0175" (UID: "7716dc96-2a29-410f-82ae-72d0d82c0175"). InnerVolumeSpecName "kube-api-access-hn69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483026    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn69c\" (UniqueName: \"kubernetes.io/projected/7716dc96-2a29-410f-82ae-72d0d82c0175-kube-api-access-hn69c\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483058    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483067    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483074    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483082    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7716dc96-2a29-410f-82ae-72d0d82c0175-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.483116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.483089    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7716dc96-2a29-410f-82ae-72d0d82c0175-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:05:51.871745 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.871650    2563 generic.go:356] "Generic (PLEG): container finished" podID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerID="947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35" exitCode=0
Sep 11 17:05:51.871745 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.871720    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"
Sep 11 17:05:51.871966 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.871738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" event={"ID":"7716dc96-2a29-410f-82ae-72d0d82c0175","Type":"ContainerDied","Data":"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"}
Sep 11 17:05:51.871966 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.871804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg" event={"ID":"7716dc96-2a29-410f-82ae-72d0d82c0175","Type":"ContainerDied","Data":"01dec0fc7341439fdebfcd8ca2b364374d415de6f2f58c023e30fdafb2228d07"}
Sep 11 17:05:51.871966 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.871837    2563 scope.go:117] "RemoveContainer" containerID="947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"
Sep 11 17:05:51.881480 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.881461    2563 scope.go:117] "RemoveContainer" containerID="947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"
Sep 11 17:05:51.881763 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:05:51.881743    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35\": container with ID starting with 947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35 not found: ID does not exist" containerID="947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"
Sep 11 17:05:51.881818 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.881773    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35"} err="failed to get container status \"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35\": rpc error: code = NotFound desc = could not find container \"947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35\": container with ID starting with 947dc7e95914ffe57cd3de897a5003091e19c99aaf10200c7a5b06ce128b8a35 not found: ID does not exist"
Sep 11 17:05:51.893431 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.893406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:05:51.897302 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:51.897281    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c99529bsg"]
Sep 11 17:05:52.042731 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:05:52.042697    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" path="/var/lib/kubelet/pods/7716dc96-2a29-410f-82ae-72d0d82c0175/volumes"
Sep 11 17:06:02.135550 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:02.135504    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:06:02.136128 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:02.135765    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy" containerID="cri-o://e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4" gracePeriod=30
Sep 11 17:06:07.389599 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.389571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:06:07.496875 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.496845    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:06:07.497186 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497172    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.497231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497187    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.497231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.497231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497206    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.497231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497225    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerName="main"
Sep 11 17:06:07.497231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497230    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerName="main"
Sep 11 17:06:07.497387 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497289    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.497387 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497296    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7716dc96-2a29-410f-82ae-72d0d82c0175" containerName="main"
Sep 11 17:06:07.497387 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.497304    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" containerName="istio-proxy"
Sep 11 17:06:07.500296 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.500278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.503305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 11 17:06:07.503416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503344    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-jz464\""
Sep 11 17:06:07.503416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 11 17:06:07.503416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503272    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 11 17:06:07.503646 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 11 17:06:07.503864 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.503847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:06:07.512761 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.512739    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:06:07.523832 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.523915 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.523915 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.523915 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd5dv\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.523972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524209 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524087    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:07.524295 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:07.524361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:06:07.524428 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524499 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data\") pod \"f326325a-b36f-465b-a518-bf121ea21b1f\" (UID: \"f326325a-b36f-465b-a518-bf121ea21b1f\") "
Sep 11 17:06:07.524710 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:07.524797 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:07.524893 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524853    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-socket\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.524893 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524876    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f326325a-b36f-465b-a518-bf121ea21b1f-istiod-ca-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.524893 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524890    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-workload-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.525066 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524903    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-credential-socket\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.525066 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.524916    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-data\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.526307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.526281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:07.526418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.526402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:06:07.526571 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.526552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:06:07.526640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.526566    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv" (OuterVolumeSpecName: "kube-api-access-sd5dv") pod "f326325a-b36f-465b-a518-bf121ea21b1f" (UID: "f326325a-b36f-465b-a518-bf121ea21b1f"). InnerVolumeSpecName "kube-api-access-sd5dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:06:07.625463 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625637 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625637 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxfg\" (UniqueName: \"kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625637 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625637 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625817    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5dv\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-kube-api-access-sd5dv\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625839    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f326325a-b36f-465b-a518-bf121ea21b1f-istio-podinfo\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.625852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625850    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f326325a-b36f-465b-a518-bf121ea21b1f-istio-envoy\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.626042 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.625859    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f326325a-b36f-465b-a518-bf121ea21b1f-istio-token\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:06:07.726418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726574 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726574 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726574 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxfg\" (UniqueName: \"kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726984 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.726984 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.726890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.727273 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.727245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.727382 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.727310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.727579 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.727558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.729025 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.729008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.729228 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.729212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.734687 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.734659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxfg\" (UniqueName: \"kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.810484 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.810455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:07.956038 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.956005    2563 generic.go:356] "Generic (PLEG): container finished" podID="f326325a-b36f-465b-a518-bf121ea21b1f" containerID="e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4" exitCode=0
Sep 11 17:06:07.956179 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.956047    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"
Sep 11 17:06:07.956179 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.956087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerDied","Data":"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"}
Sep 11 17:06:07.956179 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.956146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r" event={"ID":"f326325a-b36f-465b-a518-bf121ea21b1f","Type":"ContainerDied","Data":"1e8c92c8b01203d3f7d0a5df4600f47ff31cd84e2838dddbd1d8cc5979f23bac"}
Sep 11 17:06:07.956179 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.956170    2563 scope.go:117] "RemoveContainer" containerID="e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"
Sep 11 17:06:07.966173 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.966017    2563 scope.go:117] "RemoveContainer" containerID="6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"
Sep 11 17:06:07.973637 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.973620    2563 scope.go:117] "RemoveContainer" containerID="e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"
Sep 11 17:06:07.973939 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:06:07.973921    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4\": container with ID starting with e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4 not found: ID does not exist" containerID="e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"
Sep 11 17:06:07.974010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.973947    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4"} err="failed to get container status \"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4\": rpc error: code = NotFound desc = could not find container \"e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4\": container with ID starting with e68cb8d29fb81b37228caf7299876880998a7b08d1a8114fda3238b398c909d4 not found: ID does not exist"
Sep 11 17:06:07.974010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.973963    2563 scope.go:117] "RemoveContainer" containerID="6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"
Sep 11 17:06:07.974201 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:06:07.974186    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971\": container with ID starting with 6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971 not found: ID does not exist" containerID="6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"
Sep 11 17:06:07.974240 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.974205    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971"} err="failed to get container status \"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971\": rpc error: code = NotFound desc = could not find container \"6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971\": container with ID starting with 6763a57defb9cb714a77ed4d3dca08b4be03ec0458e6795e499b59efd9ee9971 not found: ID does not exist"
Sep 11 17:06:07.979920 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.979871    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:06:07.985454 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:07.985432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-lgp7r"]
Sep 11 17:06:08.042158 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:08.042130    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f326325a-b36f-465b-a518-bf121ea21b1f" path="/var/lib/kubelet/pods/f326325a-b36f-465b-a518-bf121ea21b1f/volumes"
Sep 11 17:06:08.092508 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:08.092486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:06:08.094392 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:06:08.094359    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3983b87_fbbc_47e6_b9fb_efa1c4f55f43.slice/crio-d90a0bf164bfed998c9ac75ca06d31022fe1647d0898207ed0a8abac00a12a22 WatchSource:0}: Error finding container d90a0bf164bfed998c9ac75ca06d31022fe1647d0898207ed0a8abac00a12a22: Status 404 returned error can't find the container with id d90a0bf164bfed998c9ac75ca06d31022fe1647d0898207ed0a8abac00a12a22
Sep 11 17:06:08.965760 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:08.965718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerStarted","Data":"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"}
Sep 11 17:06:08.966204 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:08.965791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerStarted","Data":"d90a0bf164bfed998c9ac75ca06d31022fe1647d0898207ed0a8abac00a12a22"}
Sep 11 17:06:10.979178 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:10.979145    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerID="02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8" exitCode=0
Sep 11 17:06:10.979663 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:10.979195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerDied","Data":"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"}
Sep 11 17:06:11.986431 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:11.986396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerStarted","Data":"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"}
Sep 11 17:06:12.010433 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:12.010383    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podStartSLOduration=2.818956865 podStartE2EDuration="5.01036612s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.191409255s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:06:12.007772061 +0000 UTC m=+2954.501671872" watchObservedRunningTime="2026-09-11 17:06:12.01036612 +0000 UTC m=+2954.504265932"
Sep 11 17:06:12.811347 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:12.811310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:17.811478 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:17.811442    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:06:17.812967 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:17.812941    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:06:27.811469 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:27.811417    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:06:37.810970 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:37.810923    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:06:47.811057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:47.811012    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:06:57.811461 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:57.811411    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:06:58.362233 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:58.362205    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:06:58.371647 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:06:58.371622    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:07:07.811359 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:07.811312    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:07:17.811437 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:17.811392    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:07:27.811771 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:27.811716    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:07:37.811585 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:37.811528    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:07:47.811313 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:47.811268    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:07:57.811101 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:07:57.811011    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:08:07.820705 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:07.820653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:08:07.828616 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:07.828588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:08:12.626024 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:12.625983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:08:27.648301 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:27.648253    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" containerID="cri-o://bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d" gracePeriod=45
Sep 11 17:08:27.821568 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:27.821528    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 11 17:08:28.391575 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.391550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:08:28.494715 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494772    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjxfg\" (UniqueName: \"kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494813    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.494912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.495279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home\") pod \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\" (UID: \"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43\") "
Sep 11 17:08:28.495279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.494960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:28.495279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.495184    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.495279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.495245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:08:28.495772 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.495750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home" (OuterVolumeSpecName: "home") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:28.497631 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.497607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm" (OuterVolumeSpecName: "dshm") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:28.497787 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.497709    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg" (OuterVolumeSpecName: "kube-api-access-kjxfg") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "kube-api-access-kjxfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:08:28.498033 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.498010    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:08:28.508587 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.508556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:28.518344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.518316    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" (UID: "d3983b87-fbbc-47e6-b9fb-efa1c4f55f43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:08:28.595941 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595911    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.595941 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595935    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.595941 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595944    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxfg\" (UniqueName: \"kubernetes.io/projected/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kube-api-access-kjxfg\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.596137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595955    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.596137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595964    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.596137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.595973    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.596137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.596004    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:08:28.664837 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.664801    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerID="bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d" exitCode=0
Sep 11 17:08:28.665223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.664849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"
Sep 11 17:08:28.665223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.664870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerDied","Data":"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"}
Sep 11 17:08:28.665223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.664924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd" event={"ID":"d3983b87-fbbc-47e6-b9fb-efa1c4f55f43","Type":"ContainerDied","Data":"d90a0bf164bfed998c9ac75ca06d31022fe1647d0898207ed0a8abac00a12a22"}
Sep 11 17:08:28.665223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.664948    2563 scope.go:117] "RemoveContainer" containerID="bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"
Sep 11 17:08:28.673828 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.673809    2563 scope.go:117] "RemoveContainer" containerID="02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"
Sep 11 17:08:28.681623 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.681607    2563 scope.go:117] "RemoveContainer" containerID="bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"
Sep 11 17:08:28.681929 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:08:28.681909    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d\": container with ID starting with bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d not found: ID does not exist" containerID="bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"
Sep 11 17:08:28.682015 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.681940    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d"} err="failed to get container status \"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d\": rpc error: code = NotFound desc = could not find container \"bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d\": container with ID starting with bb9bc3fd128b653d216fadc8967ff0d9f6b576fd7f41bd33aefd3778b769004d not found: ID does not exist"
Sep 11 17:08:28.682015 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.681963    2563 scope.go:117] "RemoveContainer" containerID="02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"
Sep 11 17:08:28.682238 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:08:28.682217    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8\": container with ID starting with 02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8 not found: ID does not exist" containerID="02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"
Sep 11 17:08:28.682280 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.682245    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8"} err="failed to get container status \"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8\": rpc error: code = NotFound desc = could not find container \"02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8\": container with ID starting with 02c8af7a9a511e24ef8fd6d5c335e62250404371c1747a3e7f6f37d3486f13a8 not found: ID does not exist"
Sep 11 17:08:28.687611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.687590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:08:28.692063 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:28.692042    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-589dfff8cb527nd"]
Sep 11 17:08:30.042336 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:08:30.042298    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" path="/var/lib/kubelet/pods/d3983b87-fbbc-47e6-b9fb-efa1c4f55f43/volumes"
Sep 11 17:09:04.887411 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.887377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:09:04.887959 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.887880    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="storage-initializer"
Sep 11 17:09:04.887959 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.887902    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="storage-initializer"
Sep 11 17:09:04.887959 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.887927    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main"
Sep 11 17:09:04.887959 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.887935    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main"
Sep 11 17:09:04.888177 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.888038    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3983b87-fbbc-47e6-b9fb-efa1c4f55f43" containerName="main"
Sep 11 17:09:04.892372 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.892351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:04.896074 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.896052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 11 17:09:04.896370 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.896347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 11 17:09:04.896470 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.896372    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-b88bj\""
Sep 11 17:09:04.896470 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.896355    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 11 17:09:04.902876 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:04.902854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:09:05.002142 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002142 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmm9\" (UniqueName: \"kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.002576 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.002430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103634 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103634 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103634 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103836 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmm9\" (UniqueName: \"kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103836 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103836 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.103836 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.104060 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.103873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.104158 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.104128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.105857 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.105838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.106123 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.106103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.111228 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.111209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmm9\" (UniqueName: \"kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.204406 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.204371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:05.483185 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.483159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:09:05.837333 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:05.837253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerStarted","Data":"6d4ef8ec587681635e254f7b1478ba243680abe4e2e18275095d487b5d4d5e39"}
Sep 11 17:09:14.881094 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:14.881009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerStarted","Data":"2c5c9f6c82777bcde641dfbb39aaa3b5a55533f555073ec77792114855ec6c54"}
Sep 11 17:09:15.886833 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:15.886799    2563 generic.go:356] "Generic (PLEG): container finished" podID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerID="2c5c9f6c82777bcde641dfbb39aaa3b5a55533f555073ec77792114855ec6c54" exitCode=0
Sep 11 17:09:15.887208 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:15.886883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerDied","Data":"2c5c9f6c82777bcde641dfbb39aaa3b5a55533f555073ec77792114855ec6c54"}
Sep 11 17:09:16.894736 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:16.894633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerStarted","Data":"19f32ad25802f71e0416fd97b7309ab9dc29359691111b6ebaf9d34fea8fc34d"}
Sep 11 17:09:16.894736 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:16.894710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerStarted","Data":"4b10280d16002c895265a8e11e0b161d51a05b3e5216e5c6ee0aa209a732d19c"}
Sep 11 17:09:16.916975 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:16.916931    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podStartSLOduration=2.66862131 podStartE2EDuration="12.916914694s" totalImagesPullingTime="9.06253752s" totalInitContainerRuntime="1.185755864s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:09:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:09:16.915260141 +0000 UTC m=+3139.409159954" watchObservedRunningTime="2026-09-11 17:09:16.916914694 +0000 UTC m=+3139.410814512"
Sep 11 17:09:17.205505 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:17.205478    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:25.205523 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:25.205471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:09:25.206851 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:25.206819    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:09:35.205631 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:35.205588    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:09:45.205529 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:45.205475    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:09:55.204945 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:09:55.204898    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:05.205443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:05.205395    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:15.204965 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:15.204920    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:25.205601 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:25.205545    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:35.205433 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:35.205389    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:45.205276 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:45.205231    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:10:55.205121 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:10:55.205025    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:11:05.216164 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:05.216123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:11:05.231566 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:05.231536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:11:13.811397 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:13.811358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:11:13.811843 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:13.811736    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar" containerID="cri-o://19f32ad25802f71e0416fd97b7309ab9dc29359691111b6ebaf9d34fea8fc34d" gracePeriod=60
Sep 11 17:11:14.478535 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:14.478499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh_71b4ecad-5820-4560-8d0d-a60b7a2c0f7e/modelcar/0.log"
Sep 11 17:11:14.479180 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:14.479149    2563 generic.go:356] "Generic (PLEG): container finished" podID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerID="19f32ad25802f71e0416fd97b7309ab9dc29359691111b6ebaf9d34fea8fc34d" exitCode=143
Sep 11 17:11:14.479302 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:14.479222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerDied","Data":"19f32ad25802f71e0416fd97b7309ab9dc29359691111b6ebaf9d34fea8fc34d"}
Sep 11 17:11:28.842931 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:28.842891    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" containerID="cri-o://4b10280d16002c895265a8e11e0b161d51a05b3e5216e5c6ee0aa209a732d19c" gracePeriod=45
Sep 11 17:11:29.217076 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:29.217033    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 11 17:11:29.551977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:29.551898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh_71b4ecad-5820-4560-8d0d-a60b7a2c0f7e/modelcar/0.log"
Sep 11 17:11:29.552674 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:29.552642    2563 generic.go:356] "Generic (PLEG): container finished" podID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerID="4b10280d16002c895265a8e11e0b161d51a05b3e5216e5c6ee0aa209a732d19c" exitCode=0
Sep 11 17:11:29.552803 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:29.552715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerDied","Data":"4b10280d16002c895265a8e11e0b161d51a05b3e5216e5c6ee0aa209a732d19c"}
Sep 11 17:11:30.094163 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.094140    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh_71b4ecad-5820-4560-8d0d-a60b7a2c0f7e/modelcar/0.log"
Sep 11 17:11:30.096951 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.096932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:11:30.194161 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194200    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194546 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194355    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crmm9\" (UniqueName: \"kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9\") pod \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\" (UID: \"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e\") "
Sep 11 17:11:30.194546 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:11:30.194655 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194582    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.194655 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:11:30.194864 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.194835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home" (OuterVolumeSpecName: "home") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:11:30.196450 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.196422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm" (OuterVolumeSpecName: "dshm") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:11:30.196523 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.196449    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:11:30.196523 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.196483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9" (OuterVolumeSpecName: "kube-api-access-crmm9") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "kube-api-access-crmm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:11:30.209799 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.209775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" (UID: "71b4ecad-5820-4560-8d0d-a60b7a2c0f7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:11:30.295794 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295762    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.295794 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295788    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.295794 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295798    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmm9\" (UniqueName: \"kubernetes.io/projected/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-kube-api-access-crmm9\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.296010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295806    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.296010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295814    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.296010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.295841    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:11:30.558929 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.558846    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh_71b4ecad-5820-4560-8d0d-a60b7a2c0f7e/modelcar/0.log"
Sep 11 17:11:30.559528 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.559505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh" event={"ID":"71b4ecad-5820-4560-8d0d-a60b7a2c0f7e","Type":"ContainerDied","Data":"6d4ef8ec587681635e254f7b1478ba243680abe4e2e18275095d487b5d4d5e39"}
Sep 11 17:11:30.559640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.559552    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"
Sep 11 17:11:30.559640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.559580    2563 scope.go:117] "RemoveContainer" containerID="19f32ad25802f71e0416fd97b7309ab9dc29359691111b6ebaf9d34fea8fc34d"
Sep 11 17:11:30.568723 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.568701    2563 scope.go:117] "RemoveContainer" containerID="4b10280d16002c895265a8e11e0b161d51a05b3e5216e5c6ee0aa209a732d19c"
Sep 11 17:11:30.576539 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.576521    2563 scope.go:117] "RemoveContainer" containerID="2c5c9f6c82777bcde641dfbb39aaa3b5a55533f555073ec77792114855ec6c54"
Sep 11 17:11:30.580234 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.580214    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:11:30.586068 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:30.586046    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fv25kh"]
Sep 11 17:11:32.042165 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:32.042133    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" path="/var/lib/kubelet/pods/71b4ecad-5820-4560-8d0d-a60b7a2c0f7e/volumes"
Sep 11 17:11:58.392169 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:58.392139    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:11:58.401310 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:11:58.401280    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:12:16.491559 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491520    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491876    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491896    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar-init"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491901    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar-init"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491913    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491918    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main"
Sep 11 17:12:16.491977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="main"
Sep 11 17:12:16.492198 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.491982    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b4ecad-5820-4560-8d0d-a60b7a2c0f7e" containerName="modelcar"
Sep 11 17:12:16.495108 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.495090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.497929 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.497904    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:12:16.498113 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.498056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 11 17:12:16.498222 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.498110    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 11 17:12:16.498222 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.498024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 11 17:12:16.498222 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.498057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-nt2s2\""
Sep 11 17:12:16.498222 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.497917    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 11 17:12:16.507512 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.507490    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:12:16.585755 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml686\" (UniqueName: \"kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.585755 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.585944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.585944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.585944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.585944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.586090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.586090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.585992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.686972 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.686939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.686986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687380 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml686\" (UniqueName: \"kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687380 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687489 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687489 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687606 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687734 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.687957 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.687935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.689400 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.689380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.689756 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.689741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.695356 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.695329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml686\" (UniqueName: \"kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686\") pod \"custom-route-timeout-test-kserve-5dd9d58857-v7srz\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:16.806607 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:16.806529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:17.087293 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:17.087263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:12:17.089310 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:12:17.089277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cbd1be_65cf_4add_b119_6c1d52e87806.slice/crio-1e45d7828e3d329db5a310768fa2d1bca9590cffbd8cfe7c4a7e2d5333eb42f6 WatchSource:0}: Error finding container 1e45d7828e3d329db5a310768fa2d1bca9590cffbd8cfe7c4a7e2d5333eb42f6: Status 404 returned error can't find the container with id 1e45d7828e3d329db5a310768fa2d1bca9590cffbd8cfe7c4a7e2d5333eb42f6
Sep 11 17:12:17.797852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:17.797812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerStarted","Data":"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"}
Sep 11 17:12:17.798221 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:17.797879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerStarted","Data":"1e45d7828e3d329db5a310768fa2d1bca9590cffbd8cfe7c4a7e2d5333eb42f6"}
Sep 11 17:12:19.809136 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:19.809092    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerID="a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10" exitCode=0
Sep 11 17:12:19.809528 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:19.809143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerDied","Data":"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"}
Sep 11 17:12:20.816762 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:20.816666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerStarted","Data":"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"}
Sep 11 17:12:20.840925 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:20.840870    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podStartSLOduration=2.647641632 podStartE2EDuration="4.840851617s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.193209985s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:12:20.838772581 +0000 UTC m=+3323.332672393" watchObservedRunningTime="2026-09-11 17:12:20.840851617 +0000 UTC m=+3323.334751429"
Sep 11 17:12:21.807393 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:21.807355    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:26.807634 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:26.807596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:12:26.809370 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:26.809343    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:12:36.808089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:36.808033    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:12:46.807321 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:46.807277    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:12:56.807008 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:12:56.806960    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:06.807862 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:06.807814    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:16.807858 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:16.807804    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:26.807885 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:26.807828    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:36.807467 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:36.807418    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:46.807103 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:46.807057    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:13:56.807616 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:13:56.807513    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:14:06.807471 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:06.807425    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:14:16.817589 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:16.817553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:14:16.825525 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:16.825498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:14:22.238929 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:22.238892    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:14:37.260841 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:37.260799    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" containerID="cri-o://09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec" gracePeriod=45
Sep 11 17:14:37.818541 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:37.818503    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 11 17:14:38.005538 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.005514    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:14:38.162279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162233    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162563 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162294    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml686\" (UniqueName: \"kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162563 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162563 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162563 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162360    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162803 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:14:38.162803 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm\") pod \"f9cbd1be-65cf-4add-b119-6c1d52e87806\" (UID: \"f9cbd1be-65cf-4add-b119-6c1d52e87806\") "
Sep 11 17:14:38.162910 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.162831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:14:38.163443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.163188    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbd1be-65cf-4add-b119-6c1d52e87806-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.163443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.163214    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.163443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.163432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home" (OuterVolumeSpecName: "home") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:14:38.166238 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.165115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm" (OuterVolumeSpecName: "dshm") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:14:38.172756 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.172730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686" (OuterVolumeSpecName: "kube-api-access-ml686") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "kube-api-access-ml686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:14:38.173579 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.173553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:14:38.183467 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.183442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:14:38.189915 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.189890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9cbd1be-65cf-4add-b119-6c1d52e87806" (UID: "f9cbd1be-65cf-4add-b119-6c1d52e87806"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:14:38.264670 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264629    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.264670 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264665    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml686\" (UniqueName: \"kubernetes.io/projected/f9cbd1be-65cf-4add-b119-6c1d52e87806-kube-api-access-ml686\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.264670 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264703    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cbd1be-65cf-4add-b119-6c1d52e87806-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.265143 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264715    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.265143 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264745    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.265143 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.264759    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9cbd1be-65cf-4add-b119-6c1d52e87806-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:14:38.501767 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.501733    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerID="09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec" exitCode=0
Sep 11 17:14:38.501944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.501802    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"
Sep 11 17:14:38.501944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.501818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerDied","Data":"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"}
Sep 11 17:14:38.501944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.501892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz" event={"ID":"f9cbd1be-65cf-4add-b119-6c1d52e87806","Type":"ContainerDied","Data":"1e45d7828e3d329db5a310768fa2d1bca9590cffbd8cfe7c4a7e2d5333eb42f6"}
Sep 11 17:14:38.501944 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.501914    2563 scope.go:117] "RemoveContainer" containerID="09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"
Sep 11 17:14:38.510573 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.510551    2563 scope.go:117] "RemoveContainer" containerID="a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"
Sep 11 17:14:38.517877 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.517861    2563 scope.go:117] "RemoveContainer" containerID="09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"
Sep 11 17:14:38.518113 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:14:38.518093    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec\": container with ID starting with 09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec not found: ID does not exist" containerID="09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"
Sep 11 17:14:38.518162 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.518123    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec"} err="failed to get container status \"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec\": rpc error: code = NotFound desc = could not find container \"09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec\": container with ID starting with 09a988aa38bd78c13bf4dd9260713d60313d0f16ce992c023201ee20f51410ec not found: ID does not exist"
Sep 11 17:14:38.518162 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.518139    2563 scope.go:117] "RemoveContainer" containerID="a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"
Sep 11 17:14:38.518373 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:14:38.518356    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10\": container with ID starting with a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10 not found: ID does not exist" containerID="a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"
Sep 11 17:14:38.518428 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.518381    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10"} err="failed to get container status \"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10\": rpc error: code = NotFound desc = could not find container \"a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10\": container with ID starting with a1446bd7e93ce1f57921447901de53713e885a88176050f1b9cebc82fd1c5d10 not found: ID does not exist"
Sep 11 17:14:38.525308 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.525287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:14:38.528491 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:38.528469    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-5dd9d58857-v7srz"]
Sep 11 17:14:40.041388 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:14:40.041352    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" path="/var/lib/kubelet/pods/f9cbd1be-65cf-4add-b119-6c1d52e87806/volumes"
Sep 11 17:15:12.409574 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409541    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:15:12.409958 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main"
Sep 11 17:15:12.409958 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409899    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main"
Sep 11 17:15:12.409958 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="storage-initializer"
Sep 11 17:15:12.409958 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="storage-initializer"
Sep 11 17:15:12.410100 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.409991    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cbd1be-65cf-4add-b119-6c1d52e87806" containerName="main"
Sep 11 17:15:12.412809 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.412789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.415237 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.415194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 11 17:15:12.415237 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.415224    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 11 17:15:12.415463 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.415326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-gr8rh\""
Sep 11 17:15:12.415538 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.415526    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 11 17:15:12.426898 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.426877    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:15:12.547733 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547733 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcts\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.547943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.548106 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.548106 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.547983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648643 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648643 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcts\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648870 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648870 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648870 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648870 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.648870 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649151 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649151 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.648925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649151 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.649063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649400 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.649325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649400 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.649346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649495 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.649425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.649495 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.649451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.651028 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.651006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.651373 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.651349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.656663 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.656606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.656767 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.656719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcts\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts\") pod \"router-gateway-1-openshift-default-5f4d89c756-h6sjg\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.723807 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.723772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:12.998515 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:12.998361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:15:13.000847 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:15:13.000817    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3e387b_974e_4641_afd5_3b5638b22d18.slice/crio-4204d51eebcfd58e5bf5c3f28fe82cd672e505001a3daf16a8bd92f2de7a7f0c WatchSource:0}: Error finding container 4204d51eebcfd58e5bf5c3f28fe82cd672e505001a3daf16a8bd92f2de7a7f0c: Status 404 returned error can't find the container with id 4204d51eebcfd58e5bf5c3f28fe82cd672e505001a3daf16a8bd92f2de7a7f0c
Sep 11 17:15:13.002793 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.002758    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:13.002939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.002824    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:13.002939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.002850    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:13.669350 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.669315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerStarted","Data":"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"}
Sep 11 17:15:13.669725 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.669398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerStarted","Data":"4204d51eebcfd58e5bf5c3f28fe82cd672e505001a3daf16a8bd92f2de7a7f0c"}
Sep 11 17:15:13.695864 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.695819    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podStartSLOduration=1.695804899 podStartE2EDuration="1.695804899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:13.694443224 +0000 UTC m=+3496.188343036" watchObservedRunningTime="2026-09-11 17:15:13.695804899 +0000 UTC m=+3496.189704710"
Sep 11 17:15:13.724093 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:13.724071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:14.728926 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:14.728895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:15.679184 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:15.679154    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:15.722856 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:15.722823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:22.284874 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.284836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:15:22.288943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.288918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.291617 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.291591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-v86fj\""
Sep 11 17:15:22.291776 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.291612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 11 17:15:22.291776 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.291651    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:15:22.291776 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.291714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 11 17:15:22.300233 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.300208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:15:22.433956 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.433915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.433963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434036    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjchx\" (UniqueName: \"kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434311 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.434311 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.434285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535873 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535873 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.535873 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.535722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536101 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjchx\" (UniqueName: \"kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.536344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.536333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.538216 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.538186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.538323 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.538291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.544463 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.544438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjchx\" (UniqueName: \"kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx\") pod \"router-with-refs-test-kserve-6486ccd4fc-tmkn6\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.602190 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.602152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:22.895728 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:22.895516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:15:22.898010 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:15:22.897977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd387f1f5_8260_44ed_b2ce_375b8ce3d339.slice/crio-1b21f4258dfe5f4b3d90ba348cb450fcad95afe676f2a61c8377d7c32da43f5c WatchSource:0}: Error finding container 1b21f4258dfe5f4b3d90ba348cb450fcad95afe676f2a61c8377d7c32da43f5c: Status 404 returned error can't find the container with id 1b21f4258dfe5f4b3d90ba348cb450fcad95afe676f2a61c8377d7c32da43f5c
Sep 11 17:15:23.718336 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:23.718294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerStarted","Data":"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"}
Sep 11 17:15:23.718721 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:23.718365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerStarted","Data":"1b21f4258dfe5f4b3d90ba348cb450fcad95afe676f2a61c8377d7c32da43f5c"}
Sep 11 17:15:24.725950 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.725914    2563 generic.go:356] "Generic (PLEG): container finished" podID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerID="b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c" exitCode=0
Sep 11 17:15:24.726418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.725984    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerDied","Data":"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"}
Sep 11 17:15:24.726530 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.726514    2563 scope.go:117] "RemoveContainer" containerID="b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"
Sep 11 17:15:24.727420 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.727390    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:24.727510 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.727456    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:24.727510 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:24.727484    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 17:15:25.734748 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:25.734701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerStarted","Data":"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"}
Sep 11 17:15:25.737371 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:25.737334    2563 generic.go:356] "Generic (PLEG): container finished" podID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerID="971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32" exitCode=0
Sep 11 17:15:25.737512 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:25.737440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerDied","Data":"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"}
Sep 11 17:15:26.724245 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:26.724141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:26.725449 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:26.725405    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 17:15:26.747590 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:26.747549    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerStarted","Data":"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"}
Sep 11 17:15:26.771446 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:26.771396    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podStartSLOduration=2.514674172 podStartE2EDuration="4.771379896s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.256705724s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:15:26.768633656 +0000 UTC m=+3509.262533467" watchObservedRunningTime="2026-09-11 17:15:26.771379896 +0000 UTC m=+3509.265279707"
Sep 11 17:15:27.602282 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:27.602238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:27.724281 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:27.724230    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 17:15:28.724491 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:28.724438    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 17:15:29.724919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:29.724862    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 17:15:30.679216 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:30.679179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:30.725236 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:30.725183    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 11 17:15:31.728803 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:31.728765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:31.729916 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:31.729895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:15:32.603273 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:32.603233    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:15:32.605109 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:32.605082    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:15:42.602786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:42.602739    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:15:52.603208 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:15:52.603154    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:02.603012 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:02.602964    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:12.603258 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:12.603201    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:22.602988 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:22.602940    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:32.603010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:32.602961    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:42.603631 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:42.603573    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:52.603115 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:52.603008    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:16:58.424898 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:58.424864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:16:58.438152 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:16:58.438128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:17:02.603410 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:02.603353    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:17:12.602973 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:12.602926    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:17:22.612134 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:22.612096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:17:22.619695 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:22.619654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:17:27.969620 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.969552    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:17:27.973980 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.973958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:27.977107 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977081    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 11 17:17:27.977219 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977106    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 11 17:17:27.977219 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 11 17:17:27.977372 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6xqjj\""
Sep 11 17:17:27.977466 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 11 17:17:27.977832 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.977815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:17:27.988376 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:27.988352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:17:28.161198 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gl5\" (UniqueName: \"kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.161604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.161594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.262854 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.262854 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.262854 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gl5\" (UniqueName: \"kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.262950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263288 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.263209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263556 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.263531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263700 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.263581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263700 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.263622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.263700 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.263643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.265284 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.265257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.265464 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.265444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.271560 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.271531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gl5\" (UniqueName: \"kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5\") pod \"stop-feature-test-kserve-7468ff9996-nv5d8\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.285277 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.285243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:28.572331 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:28.567576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:17:29.378405 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:29.378363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerStarted","Data":"7c1680cfe2790645c4ef0f06f7fe3606f27e2ff2654d57ea9a2b83957ed890aa"}
Sep 11 17:17:29.378844 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:29.378423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerStarted","Data":"a5b6dc056bd58c2f9aafec6f3d34383cf11b7f907feae577f35606e1d0b1777a"}
Sep 11 17:17:31.392496 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:31.392462    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerID="7c1680cfe2790645c4ef0f06f7fe3606f27e2ff2654d57ea9a2b83957ed890aa" exitCode=0
Sep 11 17:17:31.392977 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:31.392540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerDied","Data":"7c1680cfe2790645c4ef0f06f7fe3606f27e2ff2654d57ea9a2b83957ed890aa"}
Sep 11 17:17:32.399186 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:32.399152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerStarted","Data":"6d2a3bc7b56cd15695575b52259b5b46c0adfb211cd034fb880510d149343f4b"}
Sep 11 17:17:32.422081 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:32.422026    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podStartSLOduration=3.288887048 podStartE2EDuration="5.422009818s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.13312277s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:17:32.421043756 +0000 UTC m=+3634.914943572" watchObservedRunningTime="2026-09-11 17:17:32.422009818 +0000 UTC m=+3634.915909630"
Sep 11 17:17:33.285945 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:33.285901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:38.285418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:38.285371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:17:38.287312 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:38.287280    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:17:48.286232 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:48.286183    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:17:54.300207 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:54.300171    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:17:58.286051 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:17:58.286002    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:08.286422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:08.286379    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:09.323441 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:09.323395    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" containerID="cri-o://9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e" gracePeriod=45
Sep 11 17:18:09.613505 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:09.613403    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 11 17:18:10.013007 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.012977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:18:10.060778 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.060951 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.060951 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.060951 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjchx\" (UniqueName: \"kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.061194 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060964    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.061194 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.060998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.061194 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.061023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.061194 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.061090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert\") pod \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\" (UID: \"d387f1f5-8260-44ed-b2ce-375b8ce3d339\") "
Sep 11 17:18:10.061592 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.061514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache" (OuterVolumeSpecName: "model-cache") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:10.061759 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.061692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:18:10.062331 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.062309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home" (OuterVolumeSpecName: "home") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:10.064038 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.064012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:18:10.064038 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.064023    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx" (OuterVolumeSpecName: "kube-api-access-hjchx") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "kube-api-access-hjchx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:18:10.064218 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.064048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm" (OuterVolumeSpecName: "dshm") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:10.072953 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.072929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:10.087147 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.087117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d387f1f5-8260-44ed-b2ce-375b8ce3d339" (UID: "d387f1f5-8260-44ed-b2ce-375b8ce3d339"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161927    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d387f1f5-8260-44ed-b2ce-375b8ce3d339-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161956    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161965    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161974    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161985    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjchx\" (UniqueName: \"kubernetes.io/projected/d387f1f5-8260-44ed-b2ce-375b8ce3d339-kube-api-access-hjchx\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.161995    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.162004    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.162010 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.162012    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d387f1f5-8260-44ed-b2ce-375b8ce3d339-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:10.610982 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.610944    2563 generic.go:356] "Generic (PLEG): container finished" podID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerID="9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e" exitCode=0
Sep 11 17:18:10.611379 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.611014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"
Sep 11 17:18:10.611379 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.611023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerDied","Data":"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"}
Sep 11 17:18:10.611379 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.611077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6" event={"ID":"d387f1f5-8260-44ed-b2ce-375b8ce3d339","Type":"ContainerDied","Data":"1b21f4258dfe5f4b3d90ba348cb450fcad95afe676f2a61c8377d7c32da43f5c"}
Sep 11 17:18:10.611379 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.611099    2563 scope.go:117] "RemoveContainer" containerID="9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"
Sep 11 17:18:10.626399 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.626281    2563 scope.go:117] "RemoveContainer" containerID="971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"
Sep 11 17:18:10.635545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.635526    2563 scope.go:117] "RemoveContainer" containerID="9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"
Sep 11 17:18:10.635905 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:18:10.635880    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e\": container with ID starting with 9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e not found: ID does not exist" containerID="9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"
Sep 11 17:18:10.635976 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.635910    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e"} err="failed to get container status \"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e\": rpc error: code = NotFound desc = could not find container \"9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e\": container with ID starting with 9c6c387ecd9be307a3757b83d8ee24283a46098634935c8ee786ea01b7ae1b9e not found: ID does not exist"
Sep 11 17:18:10.635976 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.635927    2563 scope.go:117] "RemoveContainer" containerID="971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"
Sep 11 17:18:10.636174 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:18:10.636153    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32\": container with ID starting with 971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32 not found: ID does not exist" containerID="971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"
Sep 11 17:18:10.636215 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.636180    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32"} err="failed to get container status \"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32\": rpc error: code = NotFound desc = could not find container \"971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32\": container with ID starting with 971b76dd54117a3fbacd472c0e685396a499cba305a8d9ca22201d790cdc6d32 not found: ID does not exist"
Sep 11 17:18:10.643749 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.643722    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:18:10.649537 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:10.649512    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6486ccd4fc-tmkn6"]
Sep 11 17:18:12.043291 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:12.043251    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" path="/var/lib/kubelet/pods/d387f1f5-8260-44ed-b2ce-375b8ce3d339/volumes"
Sep 11 17:18:18.286017 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:18.285972    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:28.286767 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:28.286714    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:38.286461 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:38.286414    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:45.644947 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.644909    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:18:45.645322 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.645276    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="storage-initializer"
Sep 11 17:18:45.645322 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.645289    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="storage-initializer"
Sep 11 17:18:45.645322 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.645297    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main"
Sep 11 17:18:45.645322 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.645303    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main"
Sep 11 17:18:45.645459 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.645353    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d387f1f5-8260-44ed-b2ce-375b8ce3d339" containerName="main"
Sep 11 17:18:45.648406 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.648383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.651434 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.651407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 11 17:18:45.651548 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.651447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 11 17:18:45.651740 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.651721    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 11 17:18:45.652067 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.652048    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:18:45.652067 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.652057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4zpqq\""
Sep 11 17:18:45.653019 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.652996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 11 17:18:45.667392 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.667363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:18:45.776597 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776790 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tzp\" (UniqueName: \"kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776790 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776871 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776871 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776940 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.776940 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.777004 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.776988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877570 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877773 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877851 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877851 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877963 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.877963 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.877924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878084 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tzp\" (UniqueName: \"kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878084 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878192 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878585 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878695 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.878766 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.878701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.880890 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.880865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.881107 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.881085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.891493 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.891471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tzp\" (UniqueName: \"kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:45.960297 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:45.960260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:46.148145 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.148097    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:18:46.148370 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.148324    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" containerID="cri-o://798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e" gracePeriod=30
Sep 11 17:18:46.241539 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.241514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:18:46.246244 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:18:46.246213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f22bade_ec54_4b82_9e33_2e084906c57f.slice/crio-61b600f7082c13de348af35490e10c77fdf9a84eef45193ad058ce6069e91e2a WatchSource:0}: Error finding container 61b600f7082c13de348af35490e10c77fdf9a84eef45193ad058ce6069e91e2a: Status 404 returned error can't find the container with id 61b600f7082c13de348af35490e10c77fdf9a84eef45193ad058ce6069e91e2a
Sep 11 17:18:46.731861 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.731818    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 17:18:46.812311 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.812267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerStarted","Data":"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"}
Sep 11 17:18:46.812520 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:46.812321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerStarted","Data":"61b600f7082c13de348af35490e10c77fdf9a84eef45193ad058ce6069e91e2a"}
Sep 11 17:18:48.286324 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:48.286279    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:18:48.824212 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:48.824169    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerID="705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68" exitCode=0
Sep 11 17:18:48.824402 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:48.824237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerDied","Data":"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"}
Sep 11 17:18:49.832042 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:49.832005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerStarted","Data":"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"}
Sep 11 17:18:49.858117 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:49.858066    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podStartSLOduration=2.598544795 podStartE2EDuration="4.85804754s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.259502745s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:18:49.855547694 +0000 UTC m=+3712.349447543" watchObservedRunningTime="2026-09-11 17:18:49.85804754 +0000 UTC m=+3712.351947351"
Sep 11 17:18:51.416378 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.416315    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:18:51.529439 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529474    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529656    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529728    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:51.529919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcts\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.529919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:51.529919 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket\") pod \"4c3e387b-974e-4641-afd5-3b5638b22d18\" (UID: \"4c3e387b-974e-4641-afd5-3b5638b22d18\") "
Sep 11 17:18:51.530182 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:18:51.530182 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.529983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:51.530305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.530186    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-credential-socket\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.530305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.530213    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-socket\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.530305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.530229    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-workload-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.530305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.530243    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3e387b-974e-4641-afd5-3b5638b22d18-istiod-ca-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.530601 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.530571    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:51.532099 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.532068    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:18:51.532384 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.532358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:18:51.532457 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.532378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 17:18:51.532457 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.532421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts" (OuterVolumeSpecName: "kube-api-access-zpcts") pod "4c3e387b-974e-4641-afd5-3b5638b22d18" (UID: "4c3e387b-974e-4641-afd5-3b5638b22d18"). InnerVolumeSpecName "kube-api-access-zpcts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:18:51.631050 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.631013    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-podinfo\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.631050 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.631052    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-data\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.631254 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.631068    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-token\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.631254 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.631083    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3e387b-974e-4641-afd5-3b5638b22d18-istio-envoy\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.631254 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.631097    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcts\" (UniqueName: \"kubernetes.io/projected/4c3e387b-974e-4641-afd5-3b5638b22d18-kube-api-access-zpcts\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:18:51.844492 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.844454    2563 generic.go:356] "Generic (PLEG): container finished" podID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerID="798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e" exitCode=0
Sep 11 17:18:51.844724 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.844523    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"
Sep 11 17:18:51.844724 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.844559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerDied","Data":"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"}
Sep 11 17:18:51.844724 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.844642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg" event={"ID":"4c3e387b-974e-4641-afd5-3b5638b22d18","Type":"ContainerDied","Data":"4204d51eebcfd58e5bf5c3f28fe82cd672e505001a3daf16a8bd92f2de7a7f0c"}
Sep 11 17:18:51.844724 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.844668    2563 scope.go:117] "RemoveContainer" containerID="798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"
Sep 11 17:18:51.855932 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.855907    2563 scope.go:117] "RemoveContainer" containerID="b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"
Sep 11 17:18:51.865696 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.865614    2563 scope.go:117] "RemoveContainer" containerID="798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"
Sep 11 17:18:51.866828 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:18:51.866785    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e\": container with ID starting with 798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e not found: ID does not exist" containerID="798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"
Sep 11 17:18:51.866828 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.866819    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e"} err="failed to get container status \"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e\": rpc error: code = NotFound desc = could not find container \"798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e\": container with ID starting with 798466d47f422be74d69376dc8f153ef7e5f508ca3def238a91701d8077f392e not found: ID does not exist"
Sep 11 17:18:51.867000 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.866837    2563 scope.go:117] "RemoveContainer" containerID="b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"
Sep 11 17:18:51.867152 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:18:51.867136    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c\": container with ID starting with b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c not found: ID does not exist" containerID="b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"
Sep 11 17:18:51.867222 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.867158    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c"} err="failed to get container status \"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c\": rpc error: code = NotFound desc = could not find container \"b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c\": container with ID starting with b243c75bf77511e8797876f54d3ba105c69c86267f7402240cba67b8378cce4c not found: ID does not exist"
Sep 11 17:18:51.868193 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.868172    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:18:51.871924 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:51.871903    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5f4d89c756-h6sjg"]
Sep 11 17:18:52.043673 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:52.043628    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" path="/var/lib/kubelet/pods/4c3e387b-974e-4641-afd5-3b5638b22d18/volumes"
Sep 11 17:18:55.961465 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:55.961425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:55.961989 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:55.961507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:18:55.963229 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:55.963195    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:18:58.285772 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:18:58.285717    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:19:05.961418 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:05.961374    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:19:08.286844 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:08.286787    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:19:15.961116 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:15.961068    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:19:18.286127 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:18.286079    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 11 17:19:25.961401 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:25.961350    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:19:28.296550 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:28.296509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:19:28.304738 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:28.304705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:19:30.458988 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:30.458952    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:19:35.961405 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:35.961363    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:19:45.483280 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:45.483241    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main" containerID="cri-o://6d2a3bc7b56cd15695575b52259b5b46c0adfb211cd034fb880510d149343f4b" gracePeriod=45
Sep 11 17:19:45.961666 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:45.961623    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:19:46.139965 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.139927    2563 generic.go:356] "Generic (PLEG): container finished" podID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerID="6d2a3bc7b56cd15695575b52259b5b46c0adfb211cd034fb880510d149343f4b" exitCode=0
Sep 11 17:19:46.140137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.140008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerDied","Data":"6d2a3bc7b56cd15695575b52259b5b46c0adfb211cd034fb880510d149343f4b"}
Sep 11 17:19:46.238067 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.238036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:19:46.333241 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333411 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333264    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333411 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333411 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333359    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7gl5\" (UniqueName: \"kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location\") pod \"5bb3edd1-3423-42bc-8882-b00c18730ae2\" (UID: \"5bb3edd1-3423-42bc-8882-b00c18730ae2\") "
Sep 11 17:19:46.333859 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333810    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:19:46.333939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home" (OuterVolumeSpecName: "home") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:19:46.333996 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.333973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:19:46.336089 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.336058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm" (OuterVolumeSpecName: "dshm") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:19:46.336186 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.336127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5" (OuterVolumeSpecName: "kube-api-access-f7gl5") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "kube-api-access-f7gl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:19:46.336286 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.336268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:19:46.353364 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.353333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:19:46.355461 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.355440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bb3edd1-3423-42bc-8882-b00c18730ae2" (UID: "5bb3edd1-3423-42bc-8882-b00c18730ae2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:19:46.435165 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435129    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435165 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435157    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435165 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435169    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bb3edd1-3423-42bc-8882-b00c18730ae2-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435186    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7gl5\" (UniqueName: \"kubernetes.io/projected/5bb3edd1-3423-42bc-8882-b00c18730ae2-kube-api-access-f7gl5\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435201    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bb3edd1-3423-42bc-8882-b00c18730ae2-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435213    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435225    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:46.435374 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:46.435236    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bb3edd1-3423-42bc-8882-b00c18730ae2-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:19:47.146818 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.146789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"
Sep 11 17:19:47.146818 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.146799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8" event={"ID":"5bb3edd1-3423-42bc-8882-b00c18730ae2","Type":"ContainerDied","Data":"a5b6dc056bd58c2f9aafec6f3d34383cf11b7f907feae577f35606e1d0b1777a"}
Sep 11 17:19:47.147319 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.146878    2563 scope.go:117] "RemoveContainer" containerID="6d2a3bc7b56cd15695575b52259b5b46c0adfb211cd034fb880510d149343f4b"
Sep 11 17:19:47.156216 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.156194    2563 scope.go:117] "RemoveContainer" containerID="7c1680cfe2790645c4ef0f06f7fe3606f27e2ff2654d57ea9a2b83957ed890aa"
Sep 11 17:19:47.170734 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.170702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:19:47.173931 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:47.173905    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7468ff9996-nv5d8"]
Sep 11 17:19:48.043874 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:48.043835    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" path="/var/lib/kubelet/pods/5bb3edd1-3423-42bc-8882-b00c18730ae2/volumes"
Sep 11 17:19:55.961831 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:19:55.961728    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:20:05.961360 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:05.961308    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:20:15.961301 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:15.961247    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:20:25.961230 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:25.961178    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:20:35.961158 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:35.961103    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 11 17:20:45.971762 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:45.971730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:20:45.980015 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:45.979982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:20:56.434290 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:20:56.434254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:21:11.457058 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:11.456994    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main" containerID="cri-o://d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef" gracePeriod=45
Sep 11 17:21:12.201584 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.201561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:21:12.282024 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.281945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282024 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.281989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282190 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5tzp\" (UniqueName: \"kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282190 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282144    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282275 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282333 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282333 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282438 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home\") pod \"8f22bade-ec54-4b82-9e33-2e084906c57f\" (UID: \"8f22bade-ec54-4b82-9e33-2e084906c57f\") "
Sep 11 17:21:12.282438 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.282543 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:21:12.282607 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282597    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f22bade-ec54-4b82-9e33-2e084906c57f-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.282700 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282615    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.283001 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.282981    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home" (OuterVolumeSpecName: "home") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.284473 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.284444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm" (OuterVolumeSpecName: "dshm") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.284659 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.284641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp" (OuterVolumeSpecName: "kube-api-access-c5tzp") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "kube-api-access-c5tzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:21:12.284771 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.284740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:21:12.296231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.296201    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.304499 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.304476    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f22bade-ec54-4b82-9e33-2e084906c57f" (UID: "8f22bade-ec54-4b82-9e33-2e084906c57f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:21:12.383404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383376    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tzp\" (UniqueName: \"kubernetes.io/projected/8f22bade-ec54-4b82-9e33-2e084906c57f-kube-api-access-c5tzp\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.383404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383402    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.383545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383413    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.383545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383423    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f22bade-ec54-4b82-9e33-2e084906c57f-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.383545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383430    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.383545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.383437    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f22bade-ec54-4b82-9e33-2e084906c57f-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:21:12.559939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.559846    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerID="d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef" exitCode=0
Sep 11 17:21:12.559939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.559904    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"
Sep 11 17:21:12.559939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.559926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerDied","Data":"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"}
Sep 11 17:21:12.560443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.559983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8" event={"ID":"8f22bade-ec54-4b82-9e33-2e084906c57f","Type":"ContainerDied","Data":"61b600f7082c13de348af35490e10c77fdf9a84eef45193ad058ce6069e91e2a"}
Sep 11 17:21:12.560443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.560005    2563 scope.go:117] "RemoveContainer" containerID="d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"
Sep 11 17:21:12.569197 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.569178    2563 scope.go:117] "RemoveContainer" containerID="705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"
Sep 11 17:21:12.576638 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.576617    2563 scope.go:117] "RemoveContainer" containerID="d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"
Sep 11 17:21:12.576914 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:21:12.576895    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef\": container with ID starting with d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef not found: ID does not exist" containerID="d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"
Sep 11 17:21:12.576966 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.576923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef"} err="failed to get container status \"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef\": rpc error: code = NotFound desc = could not find container \"d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef\": container with ID starting with d15c51d5cbe5ddfaea5de8b6e84cb51a98ecfd065c0410eb4af41df3388617ef not found: ID does not exist"
Sep 11 17:21:12.576966 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.576950    2563 scope.go:117] "RemoveContainer" containerID="705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"
Sep 11 17:21:12.577216 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:21:12.577197    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68\": container with ID starting with 705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68 not found: ID does not exist" containerID="705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"
Sep 11 17:21:12.577280 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.577223    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68"} err="failed to get container status \"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68\": rpc error: code = NotFound desc = could not find container \"705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68\": container with ID starting with 705c08e05ba4e331aeec81c0911071e2af975e0cb3bcf29eaa42a6d9f3801a68 not found: ID does not exist"
Sep 11 17:21:12.582241 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.582219    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:21:12.587849 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:12.587826    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-79kmjd8"]
Sep 11 17:21:14.042148 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:14.042115    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" path="/var/lib/kubelet/pods/8f22bade-ec54-4b82-9e33-2e084906c57f/volumes"
Sep 11 17:21:58.453860 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:58.453831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:21:58.470516 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:21:58.470490    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:22:05.636985 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.636936    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:22:05.637494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637435    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.637494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637455    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637469    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="storage-initializer"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637510    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="storage-initializer"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637527    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637547    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main"
Sep 11 17:22:05.637611 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637630    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="storage-initializer"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637639    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="storage-initializer"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637755    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637773    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f22bade-ec54-4b82-9e33-2e084906c57f" containerName="main"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637787    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb3edd1-3423-42bc-8882-b00c18730ae2" containerName="main"
Sep 11 17:22:05.638054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.637963    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3e387b-974e-4641-afd5-3b5638b22d18" containerName="istio-proxy"
Sep 11 17:22:05.642317 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.642293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.645361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-nb5xl\""
Sep 11 17:22:05.645361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645153    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 11 17:22:05.645361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 11 17:22:05.645361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645250    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-pmvlv\""
Sep 11 17:22:05.645361 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:22:05.645720 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.645518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 11 17:22:05.646159 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.646141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 11 17:22:05.656523 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.656495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:22:05.724079 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724278 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724278 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724278 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724278 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724441 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724441 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zw57\" (UniqueName: \"kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.724441 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.724395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825377 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825377 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825633 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825633 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825633 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zw57\" (UniqueName: \"kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825633 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825633 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.825943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.825804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.826106 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.826077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.826172 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.826153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.826424 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.826399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.828032 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.828011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.828207 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.828189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.835746 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.835722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zw57\" (UniqueName: \"kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57\") pod \"custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:05.954390 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:05.954352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:06.233279 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:06.233255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:22:06.235243 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:22:06.235205    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481b4691_5d0b_45c2_b52c_88ec6d18186a.slice/crio-546169d0cd383db664d8ef1e1766281c482fcf04488dc839bce5d5d7f71c0b8b WatchSource:0}: Error finding container 546169d0cd383db664d8ef1e1766281c482fcf04488dc839bce5d5d7f71c0b8b: Status 404 returned error can't find the container with id 546169d0cd383db664d8ef1e1766281c482fcf04488dc839bce5d5d7f71c0b8b
Sep 11 17:22:06.237389 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:06.237366    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:22:06.818654 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:06.818614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerStarted","Data":"546169d0cd383db664d8ef1e1766281c482fcf04488dc839bce5d5d7f71c0b8b"}
Sep 11 17:22:08.829733 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:08.829674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerStarted","Data":"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"}
Sep 11 17:22:08.830217 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:08.829827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:09.839847 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:09.839803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerStarted","Data":"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"}
Sep 11 17:22:11.852674 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:11.852636    2563 generic.go:356] "Generic (PLEG): container finished" podID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerID="0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2" exitCode=0
Sep 11 17:22:11.853113 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:11.852705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerDied","Data":"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"}
Sep 11 17:22:12.860159 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:12.860121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerStarted","Data":"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"}
Sep 11 17:22:12.893174 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:12.893119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podStartSLOduration=3.654719581 podStartE2EDuration="7.893093816s" totalImagesPullingTime="2.103727482s" totalInitContainerRuntime="2.134646753s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:22:12.891175376 +0000 UTC m=+3915.385075188" watchObservedRunningTime="2026-09-11 17:22:12.893093816 +0000 UTC m=+3915.386993628"
Sep 11 17:22:15.955068 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:15.955025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:15.955490 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:15.955104    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:15.956576 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:15.956540    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:22:25.955638 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:25.955590    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:22:25.967344 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:25.967319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:22:35.955264 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:35.955208    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:22:45.955657 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:45.955600    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:22:55.955044 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:22:55.954936    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:05.955343 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:05.955289    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:15.955341 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:15.955285    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:25.955257 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:25.955207    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:35.954832 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:35.954779    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:45.955126 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:45.955065    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:23:55.955068 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:23:55.955011    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8001/health\": dial tcp 10.133.0.61:8001: connect: connection refused"
Sep 11 17:24:05.964559 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:05.964524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:24:05.977137 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:05.977106    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:24:18.156921 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:18.156887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:24:33.179393 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:33.179350    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main" containerID="cri-o://07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770" gracePeriod=45
Sep 11 17:24:33.795765 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:33.795728    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="llm-d-routing-sidecar" containerID="cri-o://41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08" gracePeriod=45
Sep 11 17:24:34.035295 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.035269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:24:34.131853 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.131814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zw57\" (UniqueName: \"kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132037 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.131875    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132037 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.131916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132037 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.131939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132037 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.131979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132037 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132317 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132050    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132317 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home\") pod \"481b4691-5d0b-45c2-b52c-88ec6d18186a\" (UID: \"481b4691-5d0b-45c2-b52c-88ec6d18186a\") "
Sep 11 17:24:34.132317 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache" (OuterVolumeSpecName: "model-cache") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.132480 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:24:34.132934 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.132905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home" (OuterVolumeSpecName: "home") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.133068 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.133052    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.133148 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.133072    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/481b4691-5d0b-45c2-b52c-88ec6d18186a-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.133148 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.133090    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.134417 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.134394    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm" (OuterVolumeSpecName: "dshm") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.134512 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.134416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57" (OuterVolumeSpecName: "kube-api-access-7zw57") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "kube-api-access-7zw57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:24:34.134512 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.134500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:24:34.149375 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.149340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.158668 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.158636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "481b4691-5d0b-45c2-b52c-88ec6d18186a" (UID: "481b4691-5d0b-45c2-b52c-88ec6d18186a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:24:34.234483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.234392    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zw57\" (UniqueName: \"kubernetes.io/projected/481b4691-5d0b-45c2-b52c-88ec6d18186a-kube-api-access-7zw57\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.234483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.234430    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481b4691-5d0b-45c2-b52c-88ec6d18186a-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.234483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.234440    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.234483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.234448    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.234483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.234456    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/481b4691-5d0b-45c2-b52c-88ec6d18186a-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:24:34.559363 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559267    2563 generic.go:356] "Generic (PLEG): container finished" podID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerID="07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770" exitCode=0
Sep 11 17:24:34.559363 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559299    2563 generic.go:356] "Generic (PLEG): container finished" podID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerID="41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08" exitCode=0
Sep 11 17:24:34.559363 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"
Sep 11 17:24:34.559363 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerDied","Data":"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"}
Sep 11 17:24:34.559712 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerDied","Data":"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"}
Sep 11 17:24:34.559712 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5" event={"ID":"481b4691-5d0b-45c2-b52c-88ec6d18186a","Type":"ContainerDied","Data":"546169d0cd383db664d8ef1e1766281c482fcf04488dc839bce5d5d7f71c0b8b"}
Sep 11 17:24:34.559712 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.559470    2563 scope.go:117] "RemoveContainer" containerID="07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"
Sep 11 17:24:34.568406 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.568382    2563 scope.go:117] "RemoveContainer" containerID="0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"
Sep 11 17:24:34.576198 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.576180    2563 scope.go:117] "RemoveContainer" containerID="41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"
Sep 11 17:24:34.583769 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.583663    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:24:34.583975 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.583957    2563 scope.go:117] "RemoveContainer" containerID="07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"
Sep 11 17:24:34.584272 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:24:34.584240    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770\": container with ID starting with 07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770 not found: ID does not exist" containerID="07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"
Sep 11 17:24:34.584343 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584283    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"} err="failed to get container status \"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770\": rpc error: code = NotFound desc = could not find container \"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770\": container with ID starting with 07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770 not found: ID does not exist"
Sep 11 17:24:34.584343 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584309    2563 scope.go:117] "RemoveContainer" containerID="0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"
Sep 11 17:24:34.584579 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:24:34.584558    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2\": container with ID starting with 0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2 not found: ID does not exist" containerID="0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"
Sep 11 17:24:34.584625 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584584    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"} err="failed to get container status \"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2\": rpc error: code = NotFound desc = could not find container \"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2\": container with ID starting with 0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2 not found: ID does not exist"
Sep 11 17:24:34.584625 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584600    2563 scope.go:117] "RemoveContainer" containerID="41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"
Sep 11 17:24:34.584861 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:24:34.584843    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08\": container with ID starting with 41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08 not found: ID does not exist" containerID="41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"
Sep 11 17:24:34.584928 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584870    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"} err="failed to get container status \"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08\": rpc error: code = NotFound desc = could not find container \"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08\": container with ID starting with 41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08 not found: ID does not exist"
Sep 11 17:24:34.584928 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.584891    2563 scope.go:117] "RemoveContainer" containerID="07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"
Sep 11 17:24:34.585153 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.585132    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770"} err="failed to get container status \"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770\": rpc error: code = NotFound desc = could not find container \"07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770\": container with ID starting with 07bd03ada1a1064605b4c66173e1be23a267d76ece720d20b56511f27cf2d770 not found: ID does not exist"
Sep 11 17:24:34.585201 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.585155    2563 scope.go:117] "RemoveContainer" containerID="0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"
Sep 11 17:24:34.585419 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.585399    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2"} err="failed to get container status \"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2\": rpc error: code = NotFound desc = could not find container \"0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2\": container with ID starting with 0e5a78a84f642e5dc628b18dbbd7ffb5ead76d8be35e9c68a5e541f9c62fd6b2 not found: ID does not exist"
Sep 11 17:24:34.585475 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.585420    2563 scope.go:117] "RemoveContainer" containerID="41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"
Sep 11 17:24:34.585656 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.585638    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08"} err="failed to get container status \"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08\": rpc error: code = NotFound desc = could not find container \"41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08\": container with ID starting with 41460d69c17c6c7b421438bb0343861f3632c679fd063a8ece4b3e67ea1a2f08 not found: ID does not exist"
Sep 11 17:24:34.589403 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:34.589381    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-b6fdfd677-wm2v5"]
Sep 11 17:24:36.042127 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:36.042093    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" path="/var/lib/kubelet/pods/481b4691-5d0b-45c2-b52c-88ec6d18186a/volumes"
Sep 11 17:24:54.372129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372094    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372416    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="storage-initializer"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372428    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="storage-initializer"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372448    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372465    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="llm-d-routing-sidecar"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372472    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="llm-d-routing-sidecar"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="main"
Sep 11 17:24:54.372542 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.372541    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="481b4691-5d0b-45c2-b52c-88ec6d18186a" containerName="llm-d-routing-sidecar"
Sep 11 17:24:54.375619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.375584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.378253 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 11 17:24:54.378382 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 11 17:24:54.378548 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-npmqg\""
Sep 11 17:24:54.378663 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:24:54.378663 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378624    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 11 17:24:54.378841 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.378673    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 11 17:24:54.388092 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.388067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:24:54.406800 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.406897 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.406897 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.407025 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.407025 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.407025 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.406974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.407025 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.407001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.407183 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.407054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skh92\" (UniqueName: \"kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508001 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.507963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508188 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508188 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508188 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508188 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508400 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508400 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skh92\" (UniqueName: \"kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508501 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508501 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.508604 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.508564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.509080 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.509061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.510499 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.510481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.510797 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.510780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.516413 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.516389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skh92\" (UniqueName: \"kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92\") pod \"lora-multiple-adapters-test-kserve-5d488cd467-5lgjg\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.686742 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.686633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:54.965836 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:54.965804    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:24:54.968543 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:24:54.968512    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920c3928_e775_4a5a_adb1_206e0b6762db.slice/crio-f76d185d1e029f2badfc64e55efbda54c93097dff70c9e52133c7404be79ac4f WatchSource:0}: Error finding container f76d185d1e029f2badfc64e55efbda54c93097dff70c9e52133c7404be79ac4f: Status 404 returned error can't find the container with id f76d185d1e029f2badfc64e55efbda54c93097dff70c9e52133c7404be79ac4f
Sep 11 17:24:55.664668 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:55.664630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerStarted","Data":"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"}
Sep 11 17:24:55.665155 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:55.664706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerStarted","Data":"f76d185d1e029f2badfc64e55efbda54c93097dff70c9e52133c7404be79ac4f"}
Sep 11 17:24:58.679486 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:58.679452    2563 generic.go:356] "Generic (PLEG): container finished" podID="920c3928-e775-4a5a-adb1-206e0b6762db" containerID="8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec" exitCode=0
Sep 11 17:24:58.679884 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:58.679526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerDied","Data":"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"}
Sep 11 17:24:59.686305 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:59.686273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerStarted","Data":"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"}
Sep 11 17:24:59.686806 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:59.686731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:24:59.710174 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:24:59.710125    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podStartSLOduration=2.500638413 podStartE2EDuration="5.71010694s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.209468527s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:24:59.707830706 +0000 UTC m=+4082.201730518" watchObservedRunningTime="2026-09-11 17:24:59.71010694 +0000 UTC m=+4082.204006752"
Sep 11 17:25:04.686833 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:04.686775    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:25:04.688470 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:04.688438    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:25:14.688111 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:14.688060    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:25:24.687702 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:24.687629    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:25:31.715070 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.715036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:25:31.718441 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.718418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.721277 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721248    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4527t\""
Sep 11 17:25:31.721277 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:25:31.721463 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721277    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 11 17:25:31.721463 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 11 17:25:31.721598 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 11 17:25:31.721653 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.721628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 11 17:25:31.731046 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.731023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:25:31.859247 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lt8h\" (UniqueName: \"kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.859798 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.859491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960732 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.960937 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961267 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.960946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lt8h\" (UniqueName: \"kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961328 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.961283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961407 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.961377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961515 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.961485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961651 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.961560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.961808 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.961641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.963339 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.963308    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.963583 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.963567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:31.968430 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:31.968376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lt8h\" (UniqueName: \"kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h\") pod \"router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:32.032023 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:32.031984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:32.317571 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:32.317544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:25:32.319482 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:25:32.319439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d90aaf_afbc_4a70_8532_e194623af6ad.slice/crio-b62732dbdbb62590ee7ab20117ab06288d32b6c1bd3567260a5aa61822a75607 WatchSource:0}: Error finding container b62732dbdbb62590ee7ab20117ab06288d32b6c1bd3567260a5aa61822a75607: Status 404 returned error can't find the container with id b62732dbdbb62590ee7ab20117ab06288d32b6c1bd3567260a5aa61822a75607
Sep 11 17:25:32.847159 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:32.847116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerStarted","Data":"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"}
Sep 11 17:25:32.847559 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:32.847180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerStarted","Data":"b62732dbdbb62590ee7ab20117ab06288d32b6c1bd3567260a5aa61822a75607"}
Sep 11 17:25:34.687660 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:34.687622    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:25:34.857898 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:34.857855    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerID="629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e" exitCode=0
Sep 11 17:25:34.858077 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:34.857921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerDied","Data":"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"}
Sep 11 17:25:35.865285 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:35.865246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerStarted","Data":"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"}
Sep 11 17:25:35.888781 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:35.888721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podStartSLOduration=2.6595041520000002 podStartE2EDuration="4.888703161s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.229199009s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:25:35.886462601 +0000 UTC m=+4118.380362408" watchObservedRunningTime="2026-09-11 17:25:35.888703161 +0000 UTC m=+4118.382602969"
Sep 11 17:25:42.032829 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:42.032791    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:42.033322 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:42.032872    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:25:42.034638 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:42.034607    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:25:44.687413 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:44.687365    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:25:52.033083 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:52.032973    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:25:54.688020 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:25:54.687974    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:26:02.032440 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:02.032381    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:04.687799 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:04.687743    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:26:12.032557 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:12.032511    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:14.697235 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:14.697196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:26:14.705504 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:14.705477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:26:22.033393 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:22.033339    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:32.033479 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:32.033420    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:42.033472 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:42.033402    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:43.315506 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:43.315463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5d488cd467-5lgjg_920c3928-e775-4a5a-adb1-206e0b6762db/storage-initializer/0.log"
Sep 11 17:26:43.324332 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:43.324305    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5d488cd467-5lgjg_920c3928-e775-4a5a-adb1-206e0b6762db/main/0.log"
Sep 11 17:26:44.371201 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:44.371167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:26:52.033231 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:52.033182    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:26:58.488702 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:58.488646    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:26:58.501813 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:58.501787    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:26:59.393472 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:59.393410    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" containerID="cri-o://5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab" gracePeriod=45
Sep 11 17:26:59.697942 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:26:59.697905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 11 17:27:00.261348 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.261322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:27:00.329916 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.329875    2563 generic.go:356] "Generic (PLEG): container finished" podID="920c3928-e775-4a5a-adb1-206e0b6762db" containerID="5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab" exitCode=0
Sep 11 17:27:00.330090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.329908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerDied","Data":"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"}
Sep 11 17:27:00.330090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.329948    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"
Sep 11 17:27:00.330090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.329988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg" event={"ID":"920c3928-e775-4a5a-adb1-206e0b6762db","Type":"ContainerDied","Data":"f76d185d1e029f2badfc64e55efbda54c93097dff70c9e52133c7404be79ac4f"}
Sep 11 17:27:00.330090 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.330021    2563 scope.go:117] "RemoveContainer" containerID="5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"
Sep 11 17:27:00.338849 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.338823    2563 scope.go:117] "RemoveContainer" containerID="8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"
Sep 11 17:27:00.347389 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.347366    2563 scope.go:117] "RemoveContainer" containerID="5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"
Sep 11 17:27:00.347646 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:27:00.347625    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab\": container with ID starting with 5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab not found: ID does not exist" containerID="5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"
Sep 11 17:27:00.347736 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.347655    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab"} err="failed to get container status \"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab\": rpc error: code = NotFound desc = could not find container \"5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab\": container with ID starting with 5c026ff454d121094ba5caae4fc95dcf92a099ad8a4043015202ffd911777dab not found: ID does not exist"
Sep 11 17:27:00.347736 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.347696    2563 scope.go:117] "RemoveContainer" containerID="8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"
Sep 11 17:27:00.347978 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:27:00.347955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec\": container with ID starting with 8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec not found: ID does not exist" containerID="8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"
Sep 11 17:27:00.348021 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.347988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec"} err="failed to get container status \"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec\": rpc error: code = NotFound desc = could not find container \"8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec\": container with ID starting with 8e73a9f9bbc7b8b82f30e3cd1de22ad53e80de08a7148a16793569a7561a2cec not found: ID does not exist"
Sep 11 17:27:00.374223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374312 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374312 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374416 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skh92\" (UniqueName: \"kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache" (OuterVolumeSpecName: "model-cache") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:00.374569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm\") pod \"920c3928-e775-4a5a-adb1-206e0b6762db\" (UID: \"920c3928-e775-4a5a-adb1-206e0b6762db\") "
Sep 11 17:27:00.374804 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home" (OuterVolumeSpecName: "home") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:00.374867 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374835    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.374867 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.374837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:27:00.376917 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.376890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:27:00.377106 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.377075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm" (OuterVolumeSpecName: "dshm") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:00.377251 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.377227    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92" (OuterVolumeSpecName: "kube-api-access-skh92") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "kube-api-access-skh92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:27:00.392309 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.392265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:00.400885 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.400852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "920c3928-e775-4a5a-adb1-206e0b6762db" (UID: "920c3928-e775-4a5a-adb1-206e0b6762db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:27:00.476018 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.475979    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920c3928-e775-4a5a-adb1-206e0b6762db-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476018 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476011    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476030    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476053    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476064    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920c3928-e775-4a5a-adb1-206e0b6762db-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476074    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skh92\" (UniqueName: \"kubernetes.io/projected/920c3928-e775-4a5a-adb1-206e0b6762db-kube-api-access-skh92\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.476214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.476081    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920c3928-e775-4a5a-adb1-206e0b6762db-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:27:00.655447 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.655414    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:27:00.660031 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:00.660006    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5d488cd467-5lgjg"]
Sep 11 17:27:02.032737 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:02.032662    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:27:02.042786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:02.042754    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" path="/var/lib/kubelet/pods/920c3928-e775-4a5a-adb1-206e0b6762db/volumes"
Sep 11 17:27:12.033360 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:12.033299    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:27:15.403479 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.403437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:27:15.404076 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.403983    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main"
Sep 11 17:27:15.404076 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.404000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main"
Sep 11 17:27:15.404076 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.404017    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="storage-initializer"
Sep 11 17:27:15.404076 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.404026    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="storage-initializer"
Sep 11 17:27:15.404290 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.404106    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="920c3928-e775-4a5a-adb1-206e0b6762db" containerName="main"
Sep 11 17:27:15.409338 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.409315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.412923 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.412890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-r67d6\""
Sep 11 17:27:15.413077 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.412915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 11 17:27:15.413077 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.412948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 11 17:27:15.413077 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.412974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 11 17:27:15.413267 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.413081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 11 17:27:15.413267 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.413257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 11 17:27:15.419613 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.419588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:27:15.496569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.496569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.496848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.496848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.496848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.496848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.497044 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.497044 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.496919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4km\" (UniqueName: \"kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.597941 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.597891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598110 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.597950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4km\" (UniqueName: \"kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598110 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.597997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598110 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598110 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598422 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598647 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598647 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.598929 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.598900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.599056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.599003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.600223 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.600202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.600596 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.600579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.609133 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.609109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4km\" (UniqueName: \"kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km\") pod \"tls-verification-test-kserve-5f7dbfd84c-vf6jr\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:15.722893 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:15.722856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:16.003846 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:16.003813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:27:16.005468 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:27:16.005440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1620dd0b_8226_441c_9cb8_17014fe21c43.slice/crio-cf751d2cc91b610fa67d00c1314abc4a5642889fab627439a039b7fabaa53cd4 WatchSource:0}: Error finding container cf751d2cc91b610fa67d00c1314abc4a5642889fab627439a039b7fabaa53cd4: Status 404 returned error can't find the container with id cf751d2cc91b610fa67d00c1314abc4a5642889fab627439a039b7fabaa53cd4
Sep 11 17:27:16.411375 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:16.411285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerStarted","Data":"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"}
Sep 11 17:27:16.411746 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:16.411385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerStarted","Data":"cf751d2cc91b610fa67d00c1314abc4a5642889fab627439a039b7fabaa53cd4"}
Sep 11 17:27:18.425626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:18.425587    2563 generic.go:356] "Generic (PLEG): container finished" podID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerID="95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506" exitCode=0
Sep 11 17:27:18.426031 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:18.425638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerDied","Data":"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"}
Sep 11 17:27:19.433337 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:19.433305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerStarted","Data":"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"}
Sep 11 17:27:19.459979 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:19.459914    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podStartSLOduration=2.162858249 podStartE2EDuration="4.459894588s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.297036339s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:27:19.456144622 +0000 UTC m=+4221.950044436" watchObservedRunningTime="2026-09-11 17:27:19.459894588 +0000 UTC m=+4221.953794405"
Sep 11 17:27:19.723202 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:19.723156    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:22.032943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:22.032844    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 11 17:27:25.724583 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:25.723729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:27:25.725765 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:25.725433    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:27:32.044610 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:32.044574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:27:32.052844 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:32.052814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:27:35.724053 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:35.724004    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:27:45.552864 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:45.552827    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:27:45.723876 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:45.723835    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:27:55.723626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:27:55.723578    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:00.578924 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:00.578877    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main" containerID="cri-o://feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b" gracePeriod=45
Sep 11 17:28:01.344002 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.343964    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:28:01.420337 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420226    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420337 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lt8h\" (UniqueName: \"kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420337 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420353    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420401    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home\") pod \"e6d90aaf-afbc-4a70-8532-e194623af6ad\" (UID: \"e6d90aaf-afbc-4a70-8532-e194623af6ad\") "
Sep 11 17:28:01.420957 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:28:01.420957 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420785    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6d90aaf-afbc-4a70-8532-e194623af6ad-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.420957 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.420885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.421397 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.421374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home" (OuterVolumeSpecName: "home") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.423003 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.422975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h" (OuterVolumeSpecName: "kube-api-access-9lt8h") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "kube-api-access-9lt8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:28:01.423519 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.423486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm" (OuterVolumeSpecName: "dshm") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.423632 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.423521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:28:01.433580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.433550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.442557 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.442531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6d90aaf-afbc-4a70-8532-e194623af6ad" (UID: "e6d90aaf-afbc-4a70-8532-e194623af6ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:28:01.521326 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521290    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521326 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521321    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6d90aaf-afbc-4a70-8532-e194623af6ad-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521326 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521330    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521551 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lt8h\" (UniqueName: \"kubernetes.io/projected/e6d90aaf-afbc-4a70-8532-e194623af6ad-kube-api-access-9lt8h\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521551 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521349    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521551 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521357    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.521551 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.521366    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6d90aaf-afbc-4a70-8532-e194623af6ad-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:28:01.657221 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.657182    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerID="feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b" exitCode=0
Sep 11 17:28:01.657672 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.657236    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"
Sep 11 17:28:01.657672 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.657261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerDied","Data":"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"}
Sep 11 17:28:01.657672 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.657347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv" event={"ID":"e6d90aaf-afbc-4a70-8532-e194623af6ad","Type":"ContainerDied","Data":"b62732dbdbb62590ee7ab20117ab06288d32b6c1bd3567260a5aa61822a75607"}
Sep 11 17:28:01.657672 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.657369    2563 scope.go:117] "RemoveContainer" containerID="feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"
Sep 11 17:28:01.667046 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.667020    2563 scope.go:117] "RemoveContainer" containerID="629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"
Sep 11 17:28:01.677028 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.677010    2563 scope.go:117] "RemoveContainer" containerID="feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"
Sep 11 17:28:01.677303 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:28:01.677282    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b\": container with ID starting with feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b not found: ID does not exist" containerID="feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"
Sep 11 17:28:01.677387 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.677315    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b"} err="failed to get container status \"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b\": rpc error: code = NotFound desc = could not find container \"feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b\": container with ID starting with feca265f7d5b37f0a492ce54c39abf5f4e79f145b3b2ca5f0d32e2831ecedd8b not found: ID does not exist"
Sep 11 17:28:01.677387 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.677339    2563 scope.go:117] "RemoveContainer" containerID="629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"
Sep 11 17:28:01.677659 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:28:01.677642    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e\": container with ID starting with 629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e not found: ID does not exist" containerID="629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"
Sep 11 17:28:01.677724 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.677665    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e"} err="failed to get container status \"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e\": rpc error: code = NotFound desc = could not find container \"629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e\": container with ID starting with 629cbf6ea8a8bfb4e5c5123dccb800f3554be03ea693be91802cf58cb119a92e not found: ID does not exist"
Sep 11 17:28:01.682221 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.682199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:28:01.686233 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:01.686208    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7987c4b879-tr9zv"]
Sep 11 17:28:02.042833 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:02.042794    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" path="/var/lib/kubelet/pods/e6d90aaf-afbc-4a70-8532-e194623af6ad/volumes"
Sep 11 17:28:05.723476 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:05.723428    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:15.724023 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:15.723968    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:25.724027 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:25.723976    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:35.723831 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:35.723778    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:45.724250 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:45.724193    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:28:55.723466 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:28:55.723362    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:29:05.724101 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:05.724045    2563 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:29:15.733833 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:15.733797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:29:15.741658 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:15.741630    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:29:20.318939 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:20.318905    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:29:35.342594 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:35.342528    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" containerID="cri-o://f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06" gracePeriod=45
Sep 11 17:29:35.734664 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:35.734612    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 11 17:29:36.085203 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.085168    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:29:36.140286 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.140252    2563 generic.go:356] "Generic (PLEG): container finished" podID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerID="f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06" exitCode=0
Sep 11 17:29:36.140456 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.140307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"
Sep 11 17:29:36.140456 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.140295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerDied","Data":"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"}
Sep 11 17:29:36.140456 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.140425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr" event={"ID":"1620dd0b-8226-441c-9cb8-17014fe21c43","Type":"ContainerDied","Data":"cf751d2cc91b610fa67d00c1314abc4a5642889fab627439a039b7fabaa53cd4"}
Sep 11 17:29:36.140456 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.140452    2563 scope.go:117] "RemoveContainer" containerID="f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"
Sep 11 17:29:36.148649 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.148630    2563 scope.go:117] "RemoveContainer" containerID="95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"
Sep 11 17:29:36.156035 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156015    2563 scope.go:117] "RemoveContainer" containerID="f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"
Sep 11 17:29:36.156284 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:29:36.156258    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06\": container with ID starting with f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06 not found: ID does not exist" containerID="f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"
Sep 11 17:29:36.156340 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06"} err="failed to get container status \"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06\": rpc error: code = NotFound desc = could not find container \"f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06\": container with ID starting with f6049feb90a42e3977ad471034c46187963ce2852e7e92b63b955725a3f7cc06 not found: ID does not exist"
Sep 11 17:29:36.156340 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156310    2563 scope.go:117] "RemoveContainer" containerID="95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"
Sep 11 17:29:36.156448 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.156448 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.156448 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.156448 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.156641 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.156721 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156659    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache" (OuterVolumeSpecName: "model-cache") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:29:36.156829 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:29:36.157032 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:29:36.157000    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506\": container with ID starting with 95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506 not found: ID does not exist" containerID="95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"
Sep 11 17:29:36.157119 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157050    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506"} err="failed to get container status \"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506\": rpc error: code = NotFound desc = could not find container \"95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506\": container with ID starting with 95be1b55aa140e34adfbbf7fb432f6818c2bc50b39b4214f521ce36a2252f506 not found: ID does not exist"
Sep 11 17:29:36.157119 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.156500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq4km\" (UniqueName: \"kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.157239 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157119    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home" (OuterVolumeSpecName: "home") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:29:36.157239 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.157398 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs\") pod \"1620dd0b-8226-441c-9cb8-17014fe21c43\" (UID: \"1620dd0b-8226-441c-9cb8-17014fe21c43\") "
Sep 11 17:29:36.157656 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157641    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.157760 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157661    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1620dd0b-8226-441c-9cb8-17014fe21c43-cabundle-cert\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.157760 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.157692    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.158749 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.158722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm" (OuterVolumeSpecName: "dshm") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:29:36.158850 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.158800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km" (OuterVolumeSpecName: "kube-api-access-vq4km") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "kube-api-access-vq4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:29:36.159664 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.159642    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:29:36.170569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.170543    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:29:36.179510 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.179488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1620dd0b-8226-441c-9cb8-17014fe21c43" (UID: "1620dd0b-8226-441c-9cb8-17014fe21c43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:29:36.258105 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.258062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4km\" (UniqueName: \"kubernetes.io/projected/1620dd0b-8226-441c-9cb8-17014fe21c43-kube-api-access-vq4km\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.258105 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.258102    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-kserve-provision-location\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.258105 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.258114    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1620dd0b-8226-441c-9cb8-17014fe21c43-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.258105 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.258123    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.258368 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.258132    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1620dd0b-8226-441c-9cb8-17014fe21c43-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:29:36.464073 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.464044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:29:36.467238 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:36.467214    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5f7dbfd84c-vf6jr"]
Sep 11 17:29:38.041406 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:29:38.041368    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" path="/var/lib/kubelet/pods/1620dd0b-8226-441c-9cb8-17014fe21c43/volumes"
Sep 11 17:30:21.684412 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:30:21.684938 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="storage-initializer"
Sep 11 17:30:21.684938 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684918    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="storage-initializer"
Sep 11 17:30:21.684938 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="storage-initializer"
Sep 11 17:30:21.685057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="storage-initializer"
Sep 11 17:30:21.685057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684959    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main"
Sep 11 17:30:21.685057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684968    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main"
Sep 11 17:30:21.685057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.684995    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main"
Sep 11 17:30:21.685057 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.685027    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main"
Sep 11 17:30:21.685246 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.685131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1620dd0b-8226-441c-9cb8-17014fe21c43" containerName="main"
Sep 11 17:30:21.685246 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.685148    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6d90aaf-afbc-4a70-8532-e194623af6ad" containerName="main"
Sep 11 17:30:21.688181 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.688159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.691848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.691819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-mzq97\""
Sep 11 17:30:21.691848 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.691830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 11 17:30:21.692014 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.691842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 11 17:30:21.692014 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.691898    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 11 17:30:21.700050 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.700027    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:30:21.843749 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.843708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtdb\" (UniqueName: \"kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.843954 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.843795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.843954 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.843897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.844081 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.843955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.844081 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.843990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.844081 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.844033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945433 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945605 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945605 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945605 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtdb\" (UniqueName: \"kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945820 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945820 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.945934 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.946016 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.946016 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.945973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.947927 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.947911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.948146 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.948129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.953483 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.953463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtdb\" (UniqueName: \"kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb\") pod \"rolling-upgrade-test-kserve-74b79d7f77-d82gt\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:21.998654 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:21.998620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:22.272168 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:22.272143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:30:22.274618 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:30:22.274579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8911b9af_1e3c_4b54_a906_b344614d8045.slice/crio-4ecc795d2e6fb5c957f0cb76721db11f2def587e37f635357b13fdedcb877536 WatchSource:0}: Error finding container 4ecc795d2e6fb5c957f0cb76721db11f2def587e37f635357b13fdedcb877536: Status 404 returned error can't find the container with id 4ecc795d2e6fb5c957f0cb76721db11f2def587e37f635357b13fdedcb877536
Sep 11 17:30:22.351921 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:22.351890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" event={"ID":"8911b9af-1e3c-4b54-a906-b344614d8045","Type":"ContainerStarted","Data":"603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33"}
Sep 11 17:30:22.352016 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:22.351943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" event={"ID":"8911b9af-1e3c-4b54-a906-b344614d8045","Type":"ContainerStarted","Data":"4ecc795d2e6fb5c957f0cb76721db11f2def587e37f635357b13fdedcb877536"}
Sep 11 17:30:23.403710 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:23.403645    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" podStartSLOduration=2.4036291419999998 podStartE2EDuration="2.403629142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:30:23.399568549 +0000 UTC m=+4405.893468361" watchObservedRunningTime="2026-09-11 17:30:23.403629142 +0000 UTC m=+4405.897528952"
Sep 11 17:30:23.999307 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:23.999273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:31.998767 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:31.998726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:32.011192 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:32.011166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:30:32.407694 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:30:32.407598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:31:13.290916 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.290876    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:31:13.299957 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:31:13.299918    2563 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 11 17:31:13.299957 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:13.299957 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:13.299957 ip-10-0-128-144 kubenswrapper[2563]:  > containerID="603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33" cmd=["/bin/sleep","15"]
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.299971    2563 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" message="" err=<
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:  >
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:31:13.299983    2563 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:         , stdout: , stderr: , exit code -1
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main" containerID="cri-o://603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33"
Sep 11 17:31:13.300225 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.300026    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main" containerID="cri-o://603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33" gracePeriod=60
Sep 11 17:31:13.397285 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.397247    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 11 17:31:13.604613 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.604579    2563 generic.go:356] "Generic (PLEG): container finished" podID="8911b9af-1e3c-4b54-a906-b344614d8045" containerID="603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33" exitCode=0
Sep 11 17:31:13.604790 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.604663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" event={"ID":"8911b9af-1e3c-4b54-a906-b344614d8045","Type":"ContainerDied","Data":"603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33"}
Sep 11 17:31:13.643362 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.643337    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:31:13.694480 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694537    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvtdb\" (UniqueName: \"kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694626 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694868 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir\") pod \"8911b9af-1e3c-4b54-a906-b344614d8045\" (UID: \"8911b9af-1e3c-4b54-a906-b344614d8045\") "
Sep 11 17:31:13.694868 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home" (OuterVolumeSpecName: "home") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:13.694981 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache" (OuterVolumeSpecName: "model-cache") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:13.694981 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694975    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-home\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:13.695073 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694981    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:13.695073 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.694992    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-model-cache\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:13.696839 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.696815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:31:13.696912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.696818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb" (OuterVolumeSpecName: "kube-api-access-mvtdb") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "kube-api-access-mvtdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:31:13.696912 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.696878    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm" (OuterVolumeSpecName: "dshm") pod "8911b9af-1e3c-4b54-a906-b344614d8045" (UID: "8911b9af-1e3c-4b54-a906-b344614d8045"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:31:13.796011 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.795981    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-dshm\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:13.796011 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.796005    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtdb\" (UniqueName: \"kubernetes.io/projected/8911b9af-1e3c-4b54-a906-b344614d8045-kube-api-access-mvtdb\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:13.796011 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.796016    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8911b9af-1e3c-4b54-a906-b344614d8045-tmp-dir\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:13.796214 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:13.796025    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8911b9af-1e3c-4b54-a906-b344614d8045-tls-certs\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:31:14.611602 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:14.611564    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"
Sep 11 17:31:14.612098 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:14.611575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt" event={"ID":"8911b9af-1e3c-4b54-a906-b344614d8045","Type":"ContainerDied","Data":"4ecc795d2e6fb5c957f0cb76721db11f2def587e37f635357b13fdedcb877536"}
Sep 11 17:31:14.612098 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:14.611733    2563 scope.go:117] "RemoveContainer" containerID="603be3e2f19679b1e48b52083e219f7c59003f87d53a34476c8849dae0241e33"
Sep 11 17:31:14.628321 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:14.628294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:31:14.632455 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:14.632431    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-d82gt"]
Sep 11 17:31:16.041824 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:16.041782    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" path="/var/lib/kubelet/pods/8911b9af-1e3c-4b54-a906-b344614d8045/volumes"
Sep 11 17:31:58.516843 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:58.516767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:31:58.533621 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:31:58.533594    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:32:13.608267 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.608229    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lww4r/must-gather-qwmr2"]
Sep 11 17:32:13.608752 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.608543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main"
Sep 11 17:32:13.608752 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.608554    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main"
Sep 11 17:32:13.608752 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.608624    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8911b9af-1e3c-4b54-a906-b344614d8045" containerName="main"
Sep 11 17:32:13.612925 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.612902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.616161 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.616139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lww4r\"/\"default-dockercfg-tqr5r\""
Sep 11 17:32:13.616283 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.616146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lww4r\"/\"kube-root-ca.crt\""
Sep 11 17:32:13.617289 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.617271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lww4r\"/\"openshift-service-ca.crt\""
Sep 11 17:32:13.621324 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.621299    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lww4r/must-gather-qwmr2"]
Sep 11 17:32:13.685737 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.685701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.685892 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.685755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzms\" (UniqueName: \"kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.786218 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.786191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.786357 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.786236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzms\" (UniqueName: \"kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.786535 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.786514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.795030 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.795005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzms\" (UniqueName: \"kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms\") pod \"must-gather-qwmr2\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") " pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:13.922510 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:13.922430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:32:14.193856 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:14.193830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lww4r/must-gather-qwmr2"]
Sep 11 17:32:14.196324 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:32:14.196290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf5f6a93_4c89_4402_a8d8_9c173b4d8809.slice/crio-a169b75826eb3f3b0443a708005116e9162fe589d9f7e346617cc3b8ee9c22ab WatchSource:0}: Error finding container a169b75826eb3f3b0443a708005116e9162fe589d9f7e346617cc3b8ee9c22ab: Status 404 returned error can't find the container with id a169b75826eb3f3b0443a708005116e9162fe589d9f7e346617cc3b8ee9c22ab
Sep 11 17:32:14.198172 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:14.198150    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:32:14.892192 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:14.892158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lww4r/must-gather-qwmr2" event={"ID":"af5f6a93-4c89-4402-a8d8-9c173b4d8809","Type":"ContainerStarted","Data":"a169b75826eb3f3b0443a708005116e9162fe589d9f7e346617cc3b8ee9c22ab"}
Sep 11 17:32:20.927580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:20.927487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lww4r/must-gather-qwmr2" event={"ID":"af5f6a93-4c89-4402-a8d8-9c173b4d8809","Type":"ContainerStarted","Data":"b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b"}
Sep 11 17:32:20.927580 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:20.927540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lww4r/must-gather-qwmr2" event={"ID":"af5f6a93-4c89-4402-a8d8-9c173b4d8809","Type":"ContainerStarted","Data":"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"}
Sep 11 17:32:20.945960 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:20.945909    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lww4r/must-gather-qwmr2" podStartSLOduration=1.656989384 podStartE2EDuration="7.945893293s" totalImagesPullingTime="6.288903909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:32:20.943489396 +0000 UTC m=+4523.437389208" watchObservedRunningTime="2026-09-11 17:32:20.945893293 +0000 UTC m=+4523.439793104"
Sep 11 17:32:39.413617 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:39.413567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:39.451051 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:39.451017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:41.076167 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:41.076110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:41.280277 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:41.280248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:43.259779 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:43.259735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:43.518054 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:43.517981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:45.168129 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:45.168067    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:45.227760 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:45.227730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:46.818294 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:46.818258    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:46.878547 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:46.878522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:48.453180 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:48.453128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:48.513532 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:48.513506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:50.113477 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:50.113430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:50.172573 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:50.172546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:51.784404 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:51.784341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:51.843163 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:51.843135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:53.401256 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:53.401209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:53.467794 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:53.467763    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:55.050926 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:55.050891    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:55.110469 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:55.110436    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:56.714105 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:56.714068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:56.774041 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:56.774009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:32:58.367250 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:58.367214    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:32:58.428762 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:32:58.428729    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:00.010415 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:00.010349    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:00.064453 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:00.064418    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:01.666052 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:01.666014    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:01.728943 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:01.728912    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:03.324534 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:03.324433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:03.388869 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:03.388838    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:04.982139 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:04.982086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:05.041714 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:05.041670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:06.602664 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:06.602623    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:06.665210 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:06.665182    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:08.237171 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:08.237131    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:08.294181 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:08.294150    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:09.881801 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:09.881765    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:09.940720 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:09.940693    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:11.495852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:11.495816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:11.553313 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:11.553282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:13.133592 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:13.133552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:13.192471 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:13.192444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:14.774738 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:14.774694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:14.837064 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:14.837037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:16.406501 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:16.406463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:16.469033 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:16.469007    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:18.038847 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:18.038810    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:18.097355 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:18.097330    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:19.697857 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:19.697820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:19.756403 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:19.756370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:21.279073 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:21.279032    2563 generic.go:356] "Generic (PLEG): container finished" podID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerID="467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad" exitCode=0
Sep 11 17:33:21.279482 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:21.279104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lww4r/must-gather-qwmr2" event={"ID":"af5f6a93-4c89-4402-a8d8-9c173b4d8809","Type":"ContainerDied","Data":"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"}
Sep 11 17:33:21.279482 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:21.279412    2563 scope.go:117] "RemoveContainer" containerID="467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"
Sep 11 17:33:21.448659 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:21.448624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lww4r_must-gather-qwmr2_af5f6a93-4c89-4402-a8d8-9c173b4d8809/gather/0.log"
Sep 11 17:33:25.184569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:25.184536    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-prnmx_ba0fbdc5-0204-4bd7-aa2a-d409b740646c/global-pull-secret-syncer/0.log"
Sep 11 17:33:25.298506 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:25.298475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f8rwn_73d5d9f0-49b4-4ac7-ab61-a3dd59301b65/konnectivity-agent/0.log"
Sep 11 17:33:25.369640 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:25.369611    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-144.ec2.internal_37310a8518ae4b4c8f438099a305077e/haproxy/0.log"
Sep 11 17:33:26.901443 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:26.901401    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lww4r/must-gather-qwmr2"]
Sep 11 17:33:26.901960 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:26.901657    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-lww4r/must-gather-qwmr2" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="copy" containerID="cri-o://b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b" gracePeriod=2
Sep 11 17:33:26.903743 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:26.903714    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lww4r/must-gather-qwmr2"]
Sep 11 17:33:26.904033 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:26.904005    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qwmr2\" is forbidden: User \"system:node:ip-10-0-128-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lww4r\": no relationship found between node 'ip-10-0-128-144.ec2.internal' and this object" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.134160 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.134134    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lww4r_must-gather-qwmr2_af5f6a93-4c89-4402-a8d8-9c173b4d8809/copy/0.log"
Sep 11 17:33:27.136789 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.136770    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.139481 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:27.139453    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qwmr2\" is forbidden: User \"system:node:ip-10-0-128-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lww4r\": no relationship found between node 'ip-10-0-128-144.ec2.internal' and this object" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.269177 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.269146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output\") pod \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") "
Sep 11 17:33:27.269366 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.269205    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjzms\" (UniqueName: \"kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms\") pod \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\" (UID: \"af5f6a93-4c89-4402-a8d8-9c173b4d8809\") "
Sep 11 17:33:27.271488 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.271455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms" (OuterVolumeSpecName: "kube-api-access-qjzms") pod "af5f6a93-4c89-4402-a8d8-9c173b4d8809" (UID: "af5f6a93-4c89-4402-a8d8-9c173b4d8809"). InnerVolumeSpecName "kube-api-access-qjzms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:33:27.276527 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.276497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "af5f6a93-4c89-4402-a8d8-9c173b4d8809" (UID: "af5f6a93-4c89-4402-a8d8-9c173b4d8809"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:33:27.309309 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.309281    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lww4r_must-gather-qwmr2_af5f6a93-4c89-4402-a8d8-9c173b4d8809/copy/0.log"
Sep 11 17:33:27.309670 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.309649    2563 generic.go:356] "Generic (PLEG): container finished" podID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerID="b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b" exitCode=143
Sep 11 17:33:27.309746 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.309717    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.309786 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.309747    2563 scope.go:117] "RemoveContainer" containerID="b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b"
Sep 11 17:33:27.313171 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:27.313142    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qwmr2\" is forbidden: User \"system:node:ip-10-0-128-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lww4r\": no relationship found between node 'ip-10-0-128-144.ec2.internal' and this object" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.317792 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.317659    2563 scope.go:117] "RemoveContainer" containerID="467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"
Sep 11 17:33:27.321504 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:27.321476    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qwmr2\" is forbidden: User \"system:node:ip-10-0-128-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lww4r\": no relationship found between node 'ip-10-0-128-144.ec2.internal' and this object" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:27.330750 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.330713    2563 scope.go:117] "RemoveContainer" containerID="b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b"
Sep 11 17:33:27.331000 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:27.330981    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b\": container with ID starting with b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b not found: ID does not exist" containerID="b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b"
Sep 11 17:33:27.331056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.331007    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b"} err="failed to get container status \"b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b\": rpc error: code = NotFound desc = could not find container \"b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b\": container with ID starting with b6aef7f77183f0db99e4854989d72619ac5171e929e036e77804f24f774f066b not found: ID does not exist"
Sep 11 17:33:27.331056 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.331028    2563 scope.go:117] "RemoveContainer" containerID="467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"
Sep 11 17:33:27.331306 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:27.331286    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad\": container with ID starting with 467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad not found: ID does not exist" containerID="467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"
Sep 11 17:33:27.331345 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.331314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad"} err="failed to get container status \"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad\": rpc error: code = NotFound desc = could not find container \"467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad\": container with ID starting with 467de8490319a3a58769c9c8e8a58e7e0554086fb5c82b035570e6f37cb3d5ad not found: ID does not exist"
Sep 11 17:33:27.369793 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.369766    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/af5f6a93-4c89-4402-a8d8-9c173b4d8809-must-gather-output\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:33:27.369793 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:27.369791    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzms\" (UniqueName: \"kubernetes.io/projected/af5f6a93-4c89-4402-a8d8-9c173b4d8809-kube-api-access-qjzms\") on node \"ip-10-0-128-144.ec2.internal\" DevicePath \"\""
Sep 11 17:33:28.040509 ip-10-0-128-144 kubenswrapper[2563]: E0911 17:33:28.040470    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-qwmr2\" is forbidden: User \"system:node:ip-10-0-128-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lww4r\": no relationship found between node 'ip-10-0-128-144.ec2.internal' and this object" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" pod="openshift-must-gather-lww4r/must-gather-qwmr2"
Sep 11 17:33:28.041716 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:28.041674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" path="/var/lib/kubelet/pods/af5f6a93-4c89-4402-a8d8-9c173b4d8809/volumes"
Sep 11 17:33:30.713623 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:30.713583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-hqmfz_1185209c-7e68-4df9-b217-0191af2b4afe/authorino/0.log"
Sep 11 17:33:30.814351 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:30.814320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-k7wvk_efe8bdb5-ecc7-41aa-ab7a-4d4f7c019dab/kuadrant-console-plugin/0.log"
Sep 11 17:33:32.539606 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:32.539546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-q5sm9_1f573c87-ea54-4c21-a781-79a4f1933a0c/monitoring-plugin/0.log"
Sep 11 17:33:32.591906 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:32.591879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9km6q_cf4ea204-cf62-4021-b549-c5dc14c5e429/node-exporter/0.log"
Sep 11 17:33:32.614454 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:32.614424    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9km6q_cf4ea204-cf62-4021-b549-c5dc14c5e429/kube-rbac-proxy/0.log"
Sep 11 17:33:32.638252 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:32.638224    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9km6q_cf4ea204-cf62-4021-b549-c5dc14c5e429/init-textfile/0.log"
Sep 11 17:33:33.070128 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.070094    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9m2z4_edba3433-9d2f-4a8d-9110-609e7aa82963/prometheus-operator/0.log"
Sep 11 17:33:33.091929 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.091904    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-9m2z4_edba3433-9d2f-4a8d-9110-609e7aa82963/kube-rbac-proxy/0.log"
Sep 11 17:33:33.120569 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.120537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-fmxkm_6a4d339c-7b66-41b2-838f-d8f6b9027896/prometheus-operator-admission-webhook/0.log"
Sep 11 17:33:33.226126 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.226090    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/thanos-query/0.log"
Sep 11 17:33:33.250540 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.250513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/kube-rbac-proxy-web/0.log"
Sep 11 17:33:33.272112 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.272088    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/kube-rbac-proxy/0.log"
Sep 11 17:33:33.293883 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.293859    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/prom-label-proxy/0.log"
Sep 11 17:33:33.316494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.316465    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/kube-rbac-proxy-rules/0.log"
Sep 11 17:33:33.338109 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:33.338037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86bb9dc985-qgvpg_58756fe8-9bbb-47cd-a724-76449798f471/kube-rbac-proxy-metrics/0.log"
Sep 11 17:33:34.469869 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:34.469835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-6zvrz_dd417906-9ea6-4cab-abd9-69af38a0132c/networking-console-plugin/0.log"
Sep 11 17:33:34.943828 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:34.943790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/2.log"
Sep 11 17:33:34.950738 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:34.950705    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-t629d_83bfe23b-dd43-457f-90f2-377a82c3d309/console-operator/3.log"
Sep 11 17:33:35.118145 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118109    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"]
Sep 11 17:33:35.118454 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="gather"
Sep 11 17:33:35.118496 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118456    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="gather"
Sep 11 17:33:35.118496 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="copy"
Sep 11 17:33:35.118496 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="copy"
Sep 11 17:33:35.118593 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118553    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="gather"
Sep 11 17:33:35.118593 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.118563    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5f6a93-4c89-4402-a8d8-9c173b4d8809" containerName="copy"
Sep 11 17:33:35.123839 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.123816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.127040 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.127019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"kube-root-ca.crt\""
Sep 11 17:33:35.127913 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.127896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvg9x\"/\"default-dockercfg-5vsk4\""
Sep 11 17:33:35.127913 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.127906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvg9x\"/\"openshift-service-ca.crt\""
Sep 11 17:33:35.132161 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.132137    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"]
Sep 11 17:33:35.237292 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.237189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zb9\" (UniqueName: \"kubernetes.io/projected/5175e14c-8af9-4861-9432-79aac55a8184-kube-api-access-r5zb9\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.237292 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.237247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-lib-modules\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.237500 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.237324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-podres\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.237500 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.237392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-sys\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.237500 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.237430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-proc\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338144 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-podres\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338316 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-sys\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338316 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-proc\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338316 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zb9\" (UniqueName: \"kubernetes.io/projected/5175e14c-8af9-4861-9432-79aac55a8184-kube-api-access-r5zb9\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338316 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-lib-modules\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338316 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-sys\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-podres\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-lib-modules\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.338494 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.338396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5175e14c-8af9-4861-9432-79aac55a8184-proc\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.346879 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.346847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zb9\" (UniqueName: \"kubernetes.io/projected/5175e14c-8af9-4861-9432-79aac55a8184-kube-api-access-r5zb9\") pod \"perf-node-gather-daemonset-vtqcd\" (UID: \"5175e14c-8af9-4861-9432-79aac55a8184\") " pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.413192 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.413151    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8468d98dcf-9z2sl_3071b390-6c46-4173-a1b5-5d4530736b42/console/0.log"
Sep 11 17:33:35.433619 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.433590    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:35.700552 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:35.700524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"]
Sep 11 17:33:35.703124 ip-10-0-128-144 kubenswrapper[2563]: W0911 17:33:35.703097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5175e14c_8af9_4861_9432_79aac55a8184.slice/crio-15df5bfad9b3f6992acd53fc4e29094d2248ca29d042ab45e987a149e7998e0e WatchSource:0}: Error finding container 15df5bfad9b3f6992acd53fc4e29094d2248ca29d042ab45e987a149e7998e0e: Status 404 returned error can't find the container with id 15df5bfad9b3f6992acd53fc4e29094d2248ca29d042ab45e987a149e7998e0e
Sep 11 17:33:36.355193 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:36.355153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd" event={"ID":"5175e14c-8af9-4861-9432-79aac55a8184","Type":"ContainerStarted","Data":"5a77eafdc19c265740bef9a5f1cf6dab252fbb54b4493c93060a65b12808cb5c"}
Sep 11 17:33:36.355380 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:36.355214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:36.355380 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:36.355235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd" event={"ID":"5175e14c-8af9-4861-9432-79aac55a8184","Type":"ContainerStarted","Data":"15df5bfad9b3f6992acd53fc4e29094d2248ca29d042ab45e987a149e7998e0e"}
Sep 11 17:33:36.371636 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:36.371585    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd" podStartSLOduration=1.371568913 podStartE2EDuration="1.371568913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:33:36.37132793 +0000 UTC m=+4598.865227743" watchObservedRunningTime="2026-09-11 17:33:36.371568913 +0000 UTC m=+4598.865468726"
Sep 11 17:33:37.037201 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.037170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hj5n_38a822e7-2ba9-4d1e-b494-1056a88c7af8/dns/0.log"
Sep 11 17:33:37.057620 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.057596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hj5n_38a822e7-2ba9-4d1e-b494-1056a88c7af8/kube-rbac-proxy/0.log"
Sep 11 17:33:37.148249 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.148220    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5nf4_72e00e16-c3b3-4c18-b68f-f491df7b9a8f/dns-node-resolver/0.log"
Sep 11 17:33:37.702643 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.702606    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-556ff8d5bf-d2h28_8f17f094-b57a-4d2f-b8f8-69cb1d19f98c/registry/0.log"
Sep 11 17:33:37.734209 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.734171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-556ff8d5bf-d2h28_8f17f094-b57a-4d2f-b8f8-69cb1d19f98c/registry/1.log"
Sep 11 17:33:37.819660 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:37.819628    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjnxd_a478cd40-13f0-40f6-8d19-12af35e82945/node-ca/0.log"
Sep 11 17:33:39.413653 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:39.413610    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qr6w_7d2d5abe-4939-4d13-97ac-78ba4b84d293/serve-healthcheck-canary/0.log"
Sep 11 17:33:39.914530 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:39.914493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-8drrk_4de17a84-cf83-4cee-a71b-d70c7cfba916/insights-operator/0.log"
Sep 11 17:33:39.918353 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:39.918326    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-8drrk_4de17a84-cf83-4cee-a71b-d70c7cfba916/insights-operator/1.log"
Sep 11 17:33:40.065994 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:40.065941    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6lpd_d5a56651-9a80-4d2a-9d3b-ab7d137682d8/kube-rbac-proxy/0.log"
Sep 11 17:33:40.088857 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:40.088829    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6lpd_d5a56651-9a80-4d2a-9d3b-ab7d137682d8/exporter/0.log"
Sep 11 17:33:40.109545 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:40.109524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h6lpd_d5a56651-9a80-4d2a-9d3b-ab7d137682d8/extractor/0.log"
Sep 11 17:33:42.369523 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:42.369496    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvg9x/perf-node-gather-daemonset-vtqcd"
Sep 11 17:33:42.981824 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:42.981797    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5956778578-5wmx6_0ab617c2-9173-429b-baa5-4d5ba3a57f84/manager/0.log"
Sep 11 17:33:43.030363 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:43.030336    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-965zk_298c6a1f-53c1-4156-81a0-95641d0c2375/openshift-lws-operator/0.log"
Sep 11 17:33:44.316301 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:44.316260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-5fcc6bb6b7-mqxx6_daec17f5-e749-4df7-a172-e19f14606f66/manager/0.log"
Sep 11 17:33:50.688218 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:50.688172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6fq7n_f9678d83-ed8e-41f3-9562-1f562984b9a2/kube-storage-version-migrator-operator/1.log"
Sep 11 17:33:50.689753 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:50.689726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-6fq7n_f9678d83-ed8e-41f3-9562-1f562984b9a2/kube-storage-version-migrator-operator/0.log"
Sep 11 17:33:51.951773 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:51.951744    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/kube-multus-additional-cni-plugins/0.log"
Sep 11 17:33:51.974459 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:51.974422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/egress-router-binary-copy/0.log"
Sep 11 17:33:52.005206 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.005177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/cni-plugins/0.log"
Sep 11 17:33:52.029091 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.029065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/bond-cni-plugin/0.log"
Sep 11 17:33:52.051205 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.051171    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/routeoverride-cni/0.log"
Sep 11 17:33:52.074412 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.074382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/whereabouts-cni-bincopy/0.log"
Sep 11 17:33:52.100828 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.100801    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rx84r_0ede9424-cc8b-4a73-b30f-8bcf98e8e129/whereabouts-cni/0.log"
Sep 11 17:33:52.195852 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.195822    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hbsz8_b9c3fd3d-faf5-4651-ba28-8b10cb47243c/kube-multus/0.log"
Sep 11 17:33:52.347599 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.347493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zjf55_13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f/network-metrics-daemon/0.log"
Sep 11 17:33:52.365093 ip-10-0-128-144 kubenswrapper[2563]: I0911 17:33:52.365063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zjf55_13d0cf25-71fd-4b8d-ab14-7c6f34f1d22f/kube-rbac-proxy/0.log"