Sep 17 17:31:42.810779 ip-10-0-128-61 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:31:42.810793 ip-10-0-128-61 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:31:42.810802 ip-10-0-128-61 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:31:42.810965 ip-10-0-128-61 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:31:53.059907 ip-10-0-128-61 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:31:53.059926 ip-10-0-128-61 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a9bb8f47b414447393ecda4eaa0fc945 --
Sep 17 17:33:43.084169 ip-10-0-128-61 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:33:43.484445 ip-10-0-128-61 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.484445 ip-10-0-128-61 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:33:43.484445 ip-10-0-128-61 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.484445 ip-10-0-128-61 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:33:43.484445 ip-10-0-128-61 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:33:43.486561 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.486460    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:33:43.490667 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490651    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.490667 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490667    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490671    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490674    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490677    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490680    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490684    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490689    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490692    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490695    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490697    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490700    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490704    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490707    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490710    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490712    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490715    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490718    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490721    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490724    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490727    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.490742 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490730    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490732    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490735    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490738    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490741    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490743    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490746    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490750    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490753    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490756    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490759    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490761    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490764    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490767    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490769    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490772    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490774    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490777    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490779    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490782    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.491325 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490784    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490787    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490789    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490792    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490794    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490797    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490800    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490802    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490805    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490807    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490810    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490812    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490815    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490820    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490825    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490828    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490831    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490834    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490838    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.491819 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490841    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490843    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490846    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490850    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490852    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490870    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490874    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490878    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490882    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490885    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490887    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490891    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490894    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490896    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490899    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490902    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490904    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490907    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490910    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.492310 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490913    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490915    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490918    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490922    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490924    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490927    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490930    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490933    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490936    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490939    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490942    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490944    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490947    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490949    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490952    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490955    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490958    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490960    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490963    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.490965    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.492765 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491677    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491691    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491694    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491697    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491702    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491706    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491710    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491714    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491718    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491721    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491723    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491726    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491729    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491731    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491734    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491737    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491740    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491742    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491745    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.493268 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491747    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491751    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491754    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491757    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491760    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491763    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491766    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491768    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491771    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491774    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491776    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491779    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491781    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491784    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491786    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491789    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491791    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491794    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491796    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491799    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.493759 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491802    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491804    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491807    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491809    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491813    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491816    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491819    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491822    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491825    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491829    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491831    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491834    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491837    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491840    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491843    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491846    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491849    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491851    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491854    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491869    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.494262 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491872    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491875    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491877    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491880    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491883    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491886    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491888    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491891    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491893    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491896    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491898    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491901    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491904    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491907    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491909    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491913    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491915    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491918    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491920    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491923    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.494757 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491925    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491928    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491931    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491933    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491936    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491939    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491942    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491944    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491947    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491950    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491952    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491955    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491958    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491960    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491963    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491965    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491968    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491970    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491973    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.491975    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.495259 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492354    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492358    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492362    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492364    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492369    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492373    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492376    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492379    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492381    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492384    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492386    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492389    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492392    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492394    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492397    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492400    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492403    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492406    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492409    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492412    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.495754 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492415    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492417    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492420    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492423    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492425    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492428    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492431    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492434    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492436    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492439    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492442    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492444    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492447    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492449    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492452    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492454    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492457    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492460    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492462    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492466    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.496273 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492468    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492470    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492474    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492478    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492481    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492484    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492486    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492489    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492492    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492495    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492497    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492501    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492504    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492506    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492509    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492513    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492516    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492519    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492522    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.496785 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492524    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492527    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492530    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492532    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492535    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492537    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492540    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492542    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492546    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492548    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492551    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492554    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492559    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492562    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492564    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492567    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492569    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492572    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492574    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492577    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.497267 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492582    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492584    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492587    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492589    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492592    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492594    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492597    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492600    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492602    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492605    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492607    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492610    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492612    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492615    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492617    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492620    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492622    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492625    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492627    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.492630    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.497784 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492707    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492740    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492747    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492751    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492759    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492762    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492767    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492772    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492775    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492778    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492782    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492785    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492790    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492793    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492797    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492800    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492803    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492806    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492811    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492814    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492817    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492821    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492824    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492828    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:33:43.498263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492831    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492834    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492838    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492841    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492844    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492847    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492851    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492854    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492878    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492893    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492896    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492899    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492903    2577 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.492909    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493632    2577 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493636    2577 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493640    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493643    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493646    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493651    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493654    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493658    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493661    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493664    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493667    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:33:43.498843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493670    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493673    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493676    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493679    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493682    2577 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493686    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493690    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493693    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493696    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493699    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493702    2577 flags.go:64] FLAG: --help="false"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493705    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493708    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493711    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493714    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493718    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493722    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493725    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493727    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493731    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493735    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493738    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493742    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493745    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:33:43.499522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493748    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493751    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493754    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493757    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493761    2577 flags.go:64] FLAG: --lock-file=""
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493764    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493767    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493771    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493776    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493779    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493782    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493784    2577 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493788    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493791    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493794    2577 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493797    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493802    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493805    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493809    2577 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493812    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493815    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493818    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493821    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493824    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493827    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:33:43.500129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493830    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493838    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493842    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493845    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493848    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493854    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493872    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493875    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493879    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493882    2577 flags.go:64] FLAG: --port="10250"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493885    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493889    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02531f3709a3caffb"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493892    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493895    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493898    2577 flags.go:64] FLAG: --register-node="true"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493901    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493904    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493907    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493910    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493913    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493917    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493920    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493923    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493926    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493929    2577 flags.go:64] FLAG: --runonce="false"
Sep 17 17:33:43.500929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493932    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493935    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493938    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493941    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493944    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493947    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493950    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493953    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493956    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493959    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493964    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493967    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493970    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493974    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493977    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493983    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493985    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493989    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493995    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.493998    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494000    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494003    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494006    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494009    2577 flags.go:64] FLAG: --v="2"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494014    2577 flags.go:64] FLAG: --version="false"
Sep 17 17:33:43.502022 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494018    2577 flags.go:64] FLAG: --vmodule=""
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494022    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494025    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494143    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494147    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494150    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494153    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494156    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494159    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494162    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494165    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494168    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494171    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494173    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494176    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494178    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494181    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494186    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494189    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494192    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.503090 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494195    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494198    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494200    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494203    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494206    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494210    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494212    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494215    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494217    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494220    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494223    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494225    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494228    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494231    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494233    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494236    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494240    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494256    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494260    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494264    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.503719 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494267    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494270    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494273    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494276    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494279    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494281    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494286    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494289    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494292    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494295    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494298    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494300    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494303    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494306    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494309    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494311    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494314    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494318    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.504270 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494320    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494324    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494327    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494330    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494333    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494336    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494339    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494341    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494344    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494346    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494349    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494351    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494354    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494357    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494359    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494362    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494364    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494367    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494369    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.505107 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494372    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494374    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494377    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494379    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494382    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494385    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494387    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494390    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494393    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494395    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494398    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494401    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494404    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494406    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494409    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494412    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494414    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494417    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494420    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494422    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.505933 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494425    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494427    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494430    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494432    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.494435    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.494442    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.503317    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.503337    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.504311    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.504331    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504433    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504441    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504447    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504452    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504457    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.506461 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504461    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504465    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504469    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504474    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504478    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504482    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504487    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504491    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504495    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504499    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504503    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504508    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504512    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504516    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504521    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504525    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504529    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504533    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504537    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504541    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.507031 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504545    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504549    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504554    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504558    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504571    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504577    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504581    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504588    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504593    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504598    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504603    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504607    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504612    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504616    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504621    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504625    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504630    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504634    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504638    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.507872 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504642    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504646    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504651    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504655    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504659    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504663    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504667    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504674    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504679    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504684    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504688    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504692    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504697    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504701    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504705    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504710    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504714    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504719    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504732    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.508531 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504737    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504742    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504746    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504750    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504754    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504761    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504767    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504772    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504776    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504781    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504786    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504791    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504795    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504800    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504804    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504808    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504812    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504816    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504820    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.509172 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504824    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504828    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504833    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504837    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504841    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504846    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504850    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504854    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504874    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504878    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504882    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504886    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504889    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504902    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504907    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504911    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.509689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.504915    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.504923    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505164    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505173    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505178    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505183    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505188    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505192    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505196    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505201    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505205    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505209    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505213    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505217    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505221    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:33:43.510121 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505225    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505229    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505233    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505238    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505242    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505246    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505251    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505255    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505259    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505263    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505267    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505272    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505276    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505281    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505285    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505297    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505301    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505306    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505310    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505314    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:33:43.510555 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505318    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505330    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505334    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505340    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505347    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505351    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505355    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505359    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505364    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505368    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505372    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505376    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505380    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505384    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505388    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505393    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505397    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505401    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505405    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505410    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:33:43.511155 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505414    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505418    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505422    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505426    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505430    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505434    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505438    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505442    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505455    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505459    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505463    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505467    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505471    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505475    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505480    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505487    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505491    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505496    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505500    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:33:43.511689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505505    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505509    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505513    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505517    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505521    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505525    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505529    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505534    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505538    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505542    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505546    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505551    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505555    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505561    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505566    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505571    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505576    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505581    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:33:43.512234 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505585    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505589    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505594    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505598    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505610    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505615    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505619    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505623    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:43.505628    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.505636    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.505832    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.509056    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.510335    2577 server.go:1019] "Starting client certificate rotation"
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.510456    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:43.512663 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.510518    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:33:43.514154 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.514136    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:43.518922 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.518890    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:33:43.529790 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.529769    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:33:43.532413 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.532397    2577 log.go:25] "Validated CRI v1 image API"
Sep 17 17:33:43.533259 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.533241    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:33:43.538968 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.538949    2577 fs.go:135] Filesystem UUIDs: map[2658f13d-90ca-4a29-a135-4337d65ee268:/dev/nvme0n1p3 27778155-4f14-47c8-8ae7-47a733b281c5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:33:43.539038 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.538968    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:33:43.544543 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.544405    2577 manager.go:216] Machine: {Timestamp:2026-09-17 17:33:43.543032917 +0000 UTC m=+0.355040845 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106045 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b03a5facd60ddfdd30d3a003259d6 SystemUUID:ec2b03a5-facd-60dd-fdd3-0d3a003259d6 BootID:a9bb8f47-b414-4473-93ec-da4eaa0fc945 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:9d:35:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:9d:35:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:1e:f1:d2:f9:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:33:43.544543 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.544537    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:33:43.544700 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.544624    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:33:43.547011 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.546994    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:43.548263 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.548239    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:33:43.548411 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.548268    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:33:43.549749 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.549736    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:33:43.549794 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.549751    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:33:43.549794 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.549759    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:43.551538 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.551527    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:33:43.553763 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.553752    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:33:43.556965 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.556952    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:43.557141 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557130    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:33:43.557178 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557175    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:33:43.557209 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557200    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:33:43.557242 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557211    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:33:43.557296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557246    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:33:43.557296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557255    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:33:43.557754 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557736    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:43.557754 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.557756    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:33:43.561427 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.561408    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:33:43.565369 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.565353    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:33:43.565446 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.565389    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567285    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567307    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567313    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567320    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567329    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:33:43.567334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567334    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567347    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567352    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567359    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567365    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567383    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:33:43.567605 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.567388    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:33:43.568422 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.568410    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:33:43.568422 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.568421    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:33:43.569630 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.569604    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:33:43.569630 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.569617    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:33:43.572306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.572286    2577 server.go:1268] "Started kubelet"
Sep 17 17:33:43.572388 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.572354    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:33:43.572457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.572385    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:33:43.572510 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.572481    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:33:43.573099 ip-10-0-128-61 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:33:43.574000 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.573984    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:33:43.576440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.576153    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:33:43.576739 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.576719    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:33:43.581496 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.581481    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:33:43.582474 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.581352    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-61.ec2.internal.18d62c215377d82e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-61.ec2.internal,UID:ip-10-0-128-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-61.ec2.internal,},FirstTimestamp:2026-09-17 17:33:43.572256814 +0000 UTC m=+0.384264740,LastTimestamp:2026-09-17 17:33:43.572256814 +0000 UTC m=+0.384264740,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-61.ec2.internal,}"
Sep 17 17:33:43.584021 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.584005    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:43.584672 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.584657    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:33:43.584875 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.584847    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:43.585467 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585448    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:33:43.585467 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585469    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:33:43.585617 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585447    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:33:43.585666 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585642    2577 factory.go:55] Registering systemd factory
Sep 17 17:33:43.585666 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585658    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:33:43.585731 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585647    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:33:43.585731 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585709    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:33:43.585921 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585906    2577 factory.go:153] Registering CRI-O factory
Sep 17 17:33:43.585988 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585925    2577 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:33:43.585988 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585976    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:33:43.586078 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.585996    2577 factory.go:103] Registering Raw factory
Sep 17 17:33:43.586078 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.586013    2577 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:33:43.586350 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.586235    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:33:43.587462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.586401    2577 manager.go:349] Starting recovery of all containers
Sep 17 17:33:43.587462 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.586322    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:33:43.599019 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.598892    2577 manager.go:354] Recovery completed
Sep 17 17:33:43.600878 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.599548    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgrpk"
Sep 17 17:33:43.604524 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.604511    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.608890 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.608870    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.608982 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.608907    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.608982 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.608921    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.609088 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.609062    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgrpk"
Sep 17 17:33:43.609694 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.609674    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:33:43.609694 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.609688    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:33:43.609694 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.609693    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.609958 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.609707    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:33:43.611479 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.611461    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.611544 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.611495    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.611544 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.611514    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.612135 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.612121    2577 policy_none.go:49] "None policy: Start"
Sep 17 17:33:43.612192 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.612139    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:33:43.612192 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.612152    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:33:43.613343 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.613327    2577 policy_none.go:47] "Start"
Sep 17 17:33:43.654624 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.654416    2577 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:33:43.654761 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.654655    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:33:43.654761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.654671    2577 server.go:87] "Starting device plugin registration server"
Sep 17 17:33:43.654918 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.654906    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:33:43.654970 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.654926    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:33:43.655255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.655015    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:33:43.655255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.655102    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:33:43.655255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.655109    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:33:43.655694 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.655666    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:33:43.655782 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.655729    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:43.689484 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.689451    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:33:43.690688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.690667    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:33:43.690776 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.690731    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:33:43.690776 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.690755    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:33:43.690854 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.690843    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:33:43.693215 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.693193    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:43.755501 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.755412    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.756540 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.756521    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.756630 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.756556    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.756630 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.756567    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.756630 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.756593    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.766010 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.765993    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.766089 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.766015    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-61.ec2.internal\": node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:43.791277 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.791252    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"]
Sep 17 17:33:43.791388 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.791325    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.793249 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.793224    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.793347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.793261    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.793347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.793275    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.793347 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.793327    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:43.795541 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.795528    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.795659 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.795643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.795708 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.795691    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.796498 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796480    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.796589 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796513    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.796589 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796527    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.796589 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796538    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.796589 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796563    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.796589 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.796576    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.798672 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.798659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.798718 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.798683    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:33:43.799511 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.799496    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:33:43.799600 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.799522    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:33:43.799600 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.799535    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:33:43.817397 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.817377    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-61.ec2.internal\" not found" node="ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.821527 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.821512    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-61.ec2.internal\" not found" node="ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.887945 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.887921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.887945 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.887947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.888113 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.887970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2516c3e83cd19c1935566a48ec58d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-61.ec2.internal\" (UID: \"d2516c3e83cd19c1935566a48ec58d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.893744 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.893719    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:43.988976 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.988950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2516c3e83cd19c1935566a48ec58d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-61.ec2.internal\" (UID: \"d2516c3e83cd19c1935566a48ec58d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.989107 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.988981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.989107 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.989002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.989107 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.989039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2516c3e83cd19c1935566a48ec58d2f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-61.ec2.internal\" (UID: \"d2516c3e83cd19c1935566a48ec58d2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.989107 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.989072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.989107 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:43.989077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7d8a1d51b151fe565decea30222bfdbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal\" (UID: \"7d8a1d51b151fe565decea30222bfdbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:43.994053 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:43.994030    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:44.094946 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:44.094814    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:44.121018 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:44.120981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:44.125059 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:44.125040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:44.195890 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:44.195843    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:44.296394 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:44.296364    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:45.550377 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:45.550349    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:45.650898 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:45.650864    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:45.662982 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.662960    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:33:45.663148 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.663131    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:33:45.751635 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:45.751606    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-61.ec2.internal\" not found"
Sep 17 17:33:45.767045 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.767019    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:45.767565 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.767534    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:28:44 +0000 UTC" deadline="2028-04-17 01:42:40.983515984 +0000 UTC"
Sep 17 17:33:45.767623 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.767566    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13856h8m55.215952965s"
Sep 17 17:33:45.838745 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.838672    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:45.853491 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.853471    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:45.854992 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.854980    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal"
Sep 17 17:33:45.865994 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.865977    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:33:45.925387 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:45.925359    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2516c3e83cd19c1935566a48ec58d2f.slice/crio-81873ead02cec963aede6d4ba3ebbb3470ed43e23fa408a6fbb4b14d930d9081 WatchSource:0}: Error finding container 81873ead02cec963aede6d4ba3ebbb3470ed43e23fa408a6fbb4b14d930d9081: Status 404 returned error can't find the container with id 81873ead02cec963aede6d4ba3ebbb3470ed43e23fa408a6fbb4b14d930d9081
Sep 17 17:33:45.926258 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:45.926231    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8a1d51b151fe565decea30222bfdbb.slice/crio-9a34a920e1358acc914c1cb3b742b620d63b51953be54094f6535aeb282cb035 WatchSource:0}: Error finding container 9a34a920e1358acc914c1cb3b742b620d63b51953be54094f6535aeb282cb035: Status 404 returned error can't find the container with id 9a34a920e1358acc914c1cb3b742b620d63b51953be54094f6535aeb282cb035
Sep 17 17:33:45.932345 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:45.932331    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:33:46.278463 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.278368    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.279555 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.279532    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:46.711408 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.711311    2577 apiserver.go:52] "Watching apiserver"
Sep 17 17:33:46.717552 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.717528    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:33:46.718906 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.718877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cfzd2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal","openshift-multus/multus-additional-cni-plugins-lg9m4","openshift-multus/multus-dq6jm","openshift-multus/network-metrics-daemon-gvfrz","kube-system/global-pull-secret-syncer-nplrw","kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5wf4z","openshift-network-diagnostics/network-check-target-v424b","openshift-network-operator/iptables-alerter-8kjp9","openshift-ovn-kubernetes/ovnkube-node-qvqqt","kube-system/konnectivity-agent-vgkv9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw","openshift-dns/node-resolver-j7xgx"]
Sep 17 17:33:46.721706 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.721437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.721706 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.721521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:46.724105 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.723555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.725737 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.725715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gjx7d\""
Sep 17 17:33:46.725829 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.725760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.725966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.725951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.727005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.726555    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:33:46.727005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.726595    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.728357 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.727944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.728783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.728648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:33:46.728783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.728679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:33:46.728783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.728678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.728783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.728719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.728783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.728730    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v5x5h\""
Sep 17 17:33:46.729726 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.729561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.729726 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.729669    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.730301 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.730250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.730644 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.730459    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:33:46.730843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.730827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jzbss\""
Sep 17 17:33:46.734296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.731954    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:33:46.734296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.732290    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:33:46.734296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.732624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wghfp\""
Sep 17 17:33:46.734296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.733244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.734296 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.733302    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:46.735447 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.735424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.737458 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.737429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.737853 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.737706    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.737853 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.737744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:46.737853 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.737809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:46.737853 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.737825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5p685\""
Sep 17 17:33:46.738369 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.738246    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:33:46.740245 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.740205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.743510 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.742246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.743510 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.742273    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:33:46.743510 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.742568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.743510 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.742948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sh2gh\""
Sep 17 17:33:46.746648 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.746629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:33:46.746740 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.746722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.747069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.747477    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.747590    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.748055    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.748232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.748753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crl2l\""
Sep 17 17:33:46.749523 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.748781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.749897 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.749826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.749946 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.749936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750082    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-bin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-kubernetes\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-kubelet-config\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1ba5e5-8de3-429e-84ae-960f72c59804-host-slash\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-modprobe-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-systemd-units\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-var-lib-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-etc-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-script-lib\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750652    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1ba5e5-8de3-429e-84ae-960f72c59804-iptables-alerter-script\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-systemd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-ovn\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovn-node-metrics-cert\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.751129 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-lib-modules\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-tmp\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.750979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-system-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-kubelet\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5vz\" (UniqueName: \"kubernetes.io/projected/5a6db059-b74f-4db5-bca9-80090a1b1fae-kube-api-access-dp5vz\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/060d1280-c356-48bc-8a89-4ddb085bf775-serviceca\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-dbus\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zl9h\" (UniqueName: \"kubernetes.io/projected/1899b110-afd4-4abf-b9bf-fdafc8354d58-kube-api-access-7zl9h\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-socket-dir-parent\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrww\" (UniqueName: \"kubernetes.io/projected/2fbabe68-32c9-4772-8316-ccc2d0c949fa-kube-api-access-vfrww\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-host\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzjp\" (UniqueName: \"kubernetes.io/projected/060d1280-c356-48bc-8a89-4ddb085bf775-kube-api-access-pvzjp\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-netns\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7p4\" (UniqueName: \"kubernetes.io/projected/6e1ba5e5-8de3-429e-84ae-960f72c59804-kube-api-access-qz7p4\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cnibin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-netns\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-log-socket\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-config\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-etc-kubernetes\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.751938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-systemd\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hnt\" (UniqueName: \"kubernetes.io/projected/9c18a0f8-b17b-4da8-82f0-e4282e527a70-kube-api-access-f7hnt\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-kubelet\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-system-cni-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-multus\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-conf-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-daemon-config\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.752966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2r69\" (UniqueName: \"kubernetes.io/projected/9d866f17-0e45-46eb-96c2-3638d706960c-kube-api-access-v2r69\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-k8s-cni-cncf-io\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/060d1280-c356-48bc-8a89-4ddb085bf775-host\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752900    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-psdff\""
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cni-binary-copy\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-hostroot\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-tuned\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-node-log\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.752996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-multus-certs\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-sys\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753085    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-slash\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cnibin\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysconfig\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-run\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.753716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-var-lib-kubelet\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-bin\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.753843    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47bfd\""
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.754586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-netd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.754683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-conf\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.754730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.754762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.754761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-env-overrides\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.755110 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.754948    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:33:46.779044 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.779022    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqgl6"
Sep 17 17:33:46.786223 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.786202    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qqgl6"
Sep 17 17:33:46.839961 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.839938    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:33:46.849491 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.849421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal" event={"ID":"d2516c3e83cd19c1935566a48ec58d2f","Type":"ContainerStarted","Data":"81873ead02cec963aede6d4ba3ebbb3470ed43e23fa408a6fbb4b14d930d9081"}
Sep 17 17:33:46.850722 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.850672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal" event={"ID":"7d8a1d51b151fe565decea30222bfdbb","Type":"ContainerStarted","Data":"9a34a920e1358acc914c1cb3b742b620d63b51953be54094f6535aeb282cb035"}
Sep 17 17:33:46.854902 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cnibin\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysconfig\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-run\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854956    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-var-lib-kubelet\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnkm2\" (UniqueName: \"kubernetes.io/projected/263b9c4f-3fea-44fe-8f45-2da654828fca-kube-api-access-mnkm2\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.854985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cnibin\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.855009 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-bin\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-netd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-run\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-conf\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/263b9c4f-3fea-44fe-8f45-2da654828fca-hosts-file\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysconfig\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-bin\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-env-overrides\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-bin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-var-lib-kubelet\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-kubernetes\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-agent-certs\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-cni-netd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855272 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-kubelet-config\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-kubelet-config\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-conf\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.855408    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1ba5e5-8de3-429e-84ae-960f72c59804-host-slash\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-modprobe-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e1ba5e5-8de3-429e-84ae-960f72c59804-host-slash\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.855491    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.355454834 +0000 UTC m=+3.013929815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-systemd-units\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-var-lib-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-kubernetes\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-modprobe-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-etc-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-etc-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-bin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.855933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-script-lib\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-var-lib-openvswitch\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-systemd-units\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1ba5e5-8de3-429e-84ae-960f72c59804-iptables-alerter-script\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbqk\" (UniqueName: \"kubernetes.io/projected/c8449501-bfe5-45cf-81ae-35547447ae4f-kube-api-access-5zbqk\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-systemd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-ovn\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-systemd\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-run-ovn\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-env-overrides\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovn-node-metrics-cert\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.855985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.856704 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-lib-modules\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-tmp\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-system-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-kubelet\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5vz\" (UniqueName: \"kubernetes.io/projected/5a6db059-b74f-4db5-bca9-80090a1b1fae-kube-api-access-dp5vz\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-sysctl-d\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-konnectivity-ca\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-device-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-script-lib\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-system-cni-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-kubelet\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e1ba5e5-8de3-429e-84ae-960f72c59804-iptables-alerter-script\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/060d1280-c356-48bc-8a89-4ddb085bf775-serviceca\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-lib-modules\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-dbus\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.857355 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zl9h\" (UniqueName: \"kubernetes.io/projected/1899b110-afd4-4abf-b9bf-fdafc8354d58-kube-api-access-7zl9h\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-socket-dir-parent\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/acebde19-4743-4eaf-9037-c00785c8e0b1-dbus\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856451    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-socket-dir-parent\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrww\" (UniqueName: \"kubernetes.io/projected/2fbabe68-32c9-4772-8316-ccc2d0c949fa-kube-api-access-vfrww\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.856712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/060d1280-c356-48bc-8a89-4ddb085bf775-serviceca\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-host\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzjp\" (UniqueName: \"kubernetes.io/projected/060d1280-c356-48bc-8a89-4ddb085bf775-kube-api-access-pvzjp\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-netns\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-host\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7p4\" (UniqueName: \"kubernetes.io/projected/6e1ba5e5-8de3-429e-84ae-960f72c59804-kube-api-access-qz7p4\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-run-netns\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cnibin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-netns\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-netns\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.858156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cnibin\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263b9c4f-3fea-44fe-8f45-2da654828fca-tmp-dir\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-log-socket\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-config\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-etc-kubernetes\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-systemd\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hnt\" (UniqueName: \"kubernetes.io/projected/9c18a0f8-b17b-4da8-82f0-e4282e527a70-kube-api-access-f7hnt\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-kubelet\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-system-cni-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-multus\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-etc-kubernetes\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-conf-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-log-socket\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-daemon-config\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-kubelet\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2r69\" (UniqueName: \"kubernetes.io/projected/9d866f17-0e45-46eb-96c2-3638d706960c-kube-api-access-v2r69\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-k8s-cni-cncf-io\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.857969    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.857987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/060d1280-c356-48bc-8a89-4ddb085bf775-host\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.858016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.358000485 +0000 UTC m=+3.016475469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cni-binary-copy\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-systemd\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-hostroot\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-tuned\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-node-log\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-multus-certs\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-sys\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.859859 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a6db059-b74f-4db5-bca9-80090a1b1fae-sys\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/060d1280-c356-48bc-8a89-4ddb085bf775-host\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-hostroot\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-cni-binary-copy\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-k8s-cni-cncf-io\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-slash\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-node-log\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-conf-dir\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-var-lib-cni-multus\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-system-cni-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-slash\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c18a0f8-b17b-4da8-82f0-e4282e527a70-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fbabe68-32c9-4772-8316-ccc2d0c949fa-host-run-multus-certs\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.858959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1899b110-afd4-4abf-b9bf-fdafc8354d58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.859106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c18a0f8-b17b-4da8-82f0-e4282e527a70-cni-binary-copy\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.860688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.859248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fbabe68-32c9-4772-8316-ccc2d0c949fa-multus-daemon-config\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.861457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.859831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovnkube-config\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.861457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.860594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-tmp\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.861457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.861254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1899b110-afd4-4abf-b9bf-fdafc8354d58-ovn-node-metrics-cert\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.861606 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.861585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a6db059-b74f-4db5-bca9-80090a1b1fae-etc-tuned\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.868403 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.868314    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:46.868403 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.868337    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:46.868403 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.868351    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:46.868599 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:46.868419    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:47.368387364 +0000 UTC m=+3.026862346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:46.870659 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.870634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrww\" (UniqueName: \"kubernetes.io/projected/2fbabe68-32c9-4772-8316-ccc2d0c949fa-kube-api-access-vfrww\") pod \"multus-dq6jm\" (UID: \"2fbabe68-32c9-4772-8316-ccc2d0c949fa\") " pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:46.871878 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.871852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zl9h\" (UniqueName: \"kubernetes.io/projected/1899b110-afd4-4abf-b9bf-fdafc8354d58-kube-api-access-7zl9h\") pod \"ovnkube-node-qvqqt\" (UID: \"1899b110-afd4-4abf-b9bf-fdafc8354d58\") " pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:46.872169 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.872147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2r69\" (UniqueName: \"kubernetes.io/projected/9d866f17-0e45-46eb-96c2-3638d706960c-kube-api-access-v2r69\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:46.872844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.872815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7p4\" (UniqueName: \"kubernetes.io/projected/6e1ba5e5-8de3-429e-84ae-960f72c59804-kube-api-access-qz7p4\") pod \"iptables-alerter-8kjp9\" (UID: \"6e1ba5e5-8de3-429e-84ae-960f72c59804\") " pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:46.873072 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.873053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hnt\" (UniqueName: \"kubernetes.io/projected/9c18a0f8-b17b-4da8-82f0-e4282e527a70-kube-api-access-f7hnt\") pod \"multus-additional-cni-plugins-lg9m4\" (UID: \"9c18a0f8-b17b-4da8-82f0-e4282e527a70\") " pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:46.873280 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.873260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzjp\" (UniqueName: \"kubernetes.io/projected/060d1280-c356-48bc-8a89-4ddb085bf775-kube-api-access-pvzjp\") pod \"node-ca-cfzd2\" (UID: \"060d1280-c356-48bc-8a89-4ddb085bf775\") " pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:46.873358 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.873339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5vz\" (UniqueName: \"kubernetes.io/projected/5a6db059-b74f-4db5-bca9-80090a1b1fae-kube-api-access-dp5vz\") pod \"tuned-5wf4z\" (UID: \"5a6db059-b74f-4db5-bca9-80090a1b1fae\") " pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:46.959344 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-konnectivity-ca\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.959534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-device-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263b9c4f-3fea-44fe-8f45-2da654828fca-tmp-dir\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.959534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnkm2\" (UniqueName: \"kubernetes.io/projected/263b9c4f-3fea-44fe-8f45-2da654828fca-kube-api-access-mnkm2\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/263b9c4f-3fea-44fe-8f45-2da654828fca-hosts-file\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-agent-certs\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959761 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbqk\" (UniqueName: \"kubernetes.io/projected/c8449501-bfe5-45cf-81ae-35547447ae4f-kube-api-access-5zbqk\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.959985 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.959951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/263b9c4f-3fea-44fe-8f45-2da654828fca-tmp-dir\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.960086 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-socket-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.960145 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.960368 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/263b9c4f-3fea-44fe-8f45-2da654828fca-hosts-file\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.960477 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-konnectivity-ca\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.960550 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-registration-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.960615 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-device-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.960671 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-sys-fs\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.960725 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.960681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c8449501-bfe5-45cf-81ae-35547447ae4f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:46.963175 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.963125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8296a988-6fdf-40e4-9b36-c2fa1b9d89e2-agent-certs\") pod \"konnectivity-agent-vgkv9\" (UID: \"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2\") " pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:46.968379 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.968346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnkm2\" (UniqueName: \"kubernetes.io/projected/263b9c4f-3fea-44fe-8f45-2da654828fca-kube-api-access-mnkm2\") pod \"node-resolver-j7xgx\" (UID: \"263b9c4f-3fea-44fe-8f45-2da654828fca\") " pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:46.968681 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:46.968658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbqk\" (UniqueName: \"kubernetes.io/projected/c8449501-bfe5-45cf-81ae-35547447ae4f-kube-api-access-5zbqk\") pod \"aws-ebs-csi-driver-node-zdwjw\" (UID: \"c8449501-bfe5-45cf-81ae-35547447ae4f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:47.037938 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.037896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kjp9"
Sep 17 17:33:47.050719 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.050695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dq6jm"
Sep 17 17:33:47.063527 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.063492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cfzd2"
Sep 17 17:33:47.068284 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.068265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lg9m4"
Sep 17 17:33:47.076889 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.076871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5wf4z"
Sep 17 17:33:47.084472 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.084452    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:33:47.092091 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.092075    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:33:47.099522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.099505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw"
Sep 17 17:33:47.105950 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.105931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7xgx"
Sep 17 17:33:47.194887 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.194853    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:33:47.362522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.362435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:47.362522 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.362494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:47.362748 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.362604    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.362748 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.362610    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.362748 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.362674    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.362654954 +0000 UTC m=+4.021129935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:47.362748 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.362694    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.362685389 +0000 UTC m=+4.021160371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:47.463084 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.463048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:47.463271 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.463243    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:47.463271 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.463270    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:47.463374 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.463284    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.463374 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:47.463351    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:48.463331735 +0000 UTC m=+4.121806732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:47.790082 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.789998    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:46 +0000 UTC" deadline="2028-03-23 18:56:18.496482698 +0000 UTC"
Sep 17 17:33:47.790082 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:47.790028    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13273h22m30.706458248s"
Sep 17 17:33:47.898493 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.898468    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8449501_bfe5_45cf_81ae_35547447ae4f.slice/crio-7e79042af19144c60ff68d0c3327fdc058587c17ade7af62cabc3cc46c4a5c04 WatchSource:0}: Error finding container 7e79042af19144c60ff68d0c3327fdc058587c17ade7af62cabc3cc46c4a5c04: Status 404 returned error can't find the container with id 7e79042af19144c60ff68d0c3327fdc058587c17ade7af62cabc3cc46c4a5c04
Sep 17 17:33:47.900582 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.900555    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1899b110_afd4_4abf_b9bf_fdafc8354d58.slice/crio-9b3e1e20411d7a126b6df6726490a4d195ef8b1f64f5dc3021b7bba20a906684 WatchSource:0}: Error finding container 9b3e1e20411d7a126b6df6726490a4d195ef8b1f64f5dc3021b7bba20a906684: Status 404 returned error can't find the container with id 9b3e1e20411d7a126b6df6726490a4d195ef8b1f64f5dc3021b7bba20a906684
Sep 17 17:33:47.902286 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.902104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6db059_b74f_4db5_bca9_80090a1b1fae.slice/crio-48cdce51a17760595f0d7e227f2af3289c879672293cc81104359e7a00368597 WatchSource:0}: Error finding container 48cdce51a17760595f0d7e227f2af3289c879672293cc81104359e7a00368597: Status 404 returned error can't find the container with id 48cdce51a17760595f0d7e227f2af3289c879672293cc81104359e7a00368597
Sep 17 17:33:47.904805 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.904781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbabe68_32c9_4772_8316_ccc2d0c949fa.slice/crio-a38429e5ddd4b3f024d173926cceaa4c5e5575b2d99cadd42c4f752ab8e5ba63 WatchSource:0}: Error finding container a38429e5ddd4b3f024d173926cceaa4c5e5575b2d99cadd42c4f752ab8e5ba63: Status 404 returned error can't find the container with id a38429e5ddd4b3f024d173926cceaa4c5e5575b2d99cadd42c4f752ab8e5ba63
Sep 17 17:33:47.926972 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.926948    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1ba5e5_8de3_429e_84ae_960f72c59804.slice/crio-47f8cba22d2dbcf9825b8ec0e4cc3689833fa8e1279f4b5310eead9c151cb04b WatchSource:0}: Error finding container 47f8cba22d2dbcf9825b8ec0e4cc3689833fa8e1279f4b5310eead9c151cb04b: Status 404 returned error can't find the container with id 47f8cba22d2dbcf9825b8ec0e4cc3689833fa8e1279f4b5310eead9c151cb04b
Sep 17 17:33:47.927710 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:33:47.927689    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c18a0f8_b17b_4da8_82f0_e4282e527a70.slice/crio-d0ce9c3bdee208bbcda32fcdbedffc93c2dc991501b19b531011d492dbc36c77 WatchSource:0}: Error finding container d0ce9c3bdee208bbcda32fcdbedffc93c2dc991501b19b531011d492dbc36c77: Status 404 returned error can't find the container with id d0ce9c3bdee208bbcda32fcdbedffc93c2dc991501b19b531011d492dbc36c77
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.368889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.369162    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.369184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.369235    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.369208189 +0000 UTC m=+6.027683167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.369276    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.369752 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.369312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.369301911 +0000 UTC m=+6.027776905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:48.470169 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.470094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:48.470320 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.470288    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:48.470320 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.470306    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:48.470320 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.470317    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.470528 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.470373    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:50.470355114 +0000 UTC m=+6.128830094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:48.790934 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.790796    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:28:46 +0000 UTC" deadline="2028-06-10 23:05:55.038130526 +0000 UTC"
Sep 17 17:33:48.790934 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.790857    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15173h32m6.247276893s"
Sep 17 17:33:48.847237 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.847207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:48.847414 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.847342    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:48.847763 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.847741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:48.847864 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.847844    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:48.847931 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.847910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:48.848004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:48.847987    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:48.880925 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.880893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgkv9" event={"ID":"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2","Type":"ContainerStarted","Data":"cca52a21bc34e71126b905befdbee0f22271cc700f02cb4f7dab5a57e5590d92"}
Sep 17 17:33:48.891952 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.891893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7xgx" event={"ID":"263b9c4f-3fea-44fe-8f45-2da654828fca","Type":"ContainerStarted","Data":"7fa873cd3c26c6f3db9e90043cdb6d4c995838e92efb2e07e95062d3840df2e5"}
Sep 17 17:33:48.900250 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.900220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kjp9" event={"ID":"6e1ba5e5-8de3-429e-84ae-960f72c59804","Type":"ContainerStarted","Data":"47f8cba22d2dbcf9825b8ec0e4cc3689833fa8e1279f4b5310eead9c151cb04b"}
Sep 17 17:33:48.904831 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.904762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5wf4z" event={"ID":"5a6db059-b74f-4db5-bca9-80090a1b1fae","Type":"ContainerStarted","Data":"48cdce51a17760595f0d7e227f2af3289c879672293cc81104359e7a00368597"}
Sep 17 17:33:48.907743 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.907663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw" event={"ID":"c8449501-bfe5-45cf-81ae-35547447ae4f","Type":"ContainerStarted","Data":"7e79042af19144c60ff68d0c3327fdc058587c17ade7af62cabc3cc46c4a5c04"}
Sep 17 17:33:48.917687 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.916924    2577 generic.go:356] "Generic (PLEG): container finished" podID="7d8a1d51b151fe565decea30222bfdbb" containerID="74b296a1fe3783191efac4474633b1c65d7faaaa01338ee9748809ac96c2e9e9" exitCode=0
Sep 17 17:33:48.917687 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.916998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal" event={"ID":"7d8a1d51b151fe565decea30222bfdbb","Type":"ContainerDied","Data":"74b296a1fe3783191efac4474633b1c65d7faaaa01338ee9748809ac96c2e9e9"}
Sep 17 17:33:48.937124 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.937008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfzd2" event={"ID":"060d1280-c356-48bc-8a89-4ddb085bf775","Type":"ContainerStarted","Data":"90db97b6b4503d423a5111100fce8442bfc3511b8a88c34e03b2075ea816e3d1"}
Sep 17 17:33:48.955650 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.954899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerStarted","Data":"d0ce9c3bdee208bbcda32fcdbedffc93c2dc991501b19b531011d492dbc36c77"}
Sep 17 17:33:48.968023 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.967919    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dq6jm" event={"ID":"2fbabe68-32c9-4772-8316-ccc2d0c949fa","Type":"ContainerStarted","Data":"a38429e5ddd4b3f024d173926cceaa4c5e5575b2d99cadd42c4f752ab8e5ba63"}
Sep 17 17:33:48.973792 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:48.973768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"9b3e1e20411d7a126b6df6726490a4d195ef8b1f64f5dc3021b7bba20a906684"}
Sep 17 17:33:49.003351 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:49.003324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal" event={"ID":"d2516c3e83cd19c1935566a48ec58d2f","Type":"ContainerStarted","Data":"094b7dc6d12de890ebddcdd46d05a473821628024f7756c8f72f99e46dcf38f0"}
Sep 17 17:33:49.019561 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:49.019281    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-61.ec2.internal" podStartSLOduration=4.019264069 podStartE2EDuration="4.019264069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:49.018361144 +0000 UTC m=+4.676836145" watchObservedRunningTime="2026-09-17 17:33:49.019264069 +0000 UTC m=+4.677739067"
Sep 17 17:33:50.031044 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.030965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal" event={"ID":"7d8a1d51b151fe565decea30222bfdbb","Type":"ContainerStarted","Data":"efba0fe1970583d41059302542584974448d6aeca64cd733cb0030f80ae53802"}
Sep 17 17:33:50.047681 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.047631    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-61.ec2.internal" podStartSLOduration=5.047612311 podStartE2EDuration="5.047612311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:33:50.047415507 +0000 UTC m=+5.705890504" watchObservedRunningTime="2026-09-17 17:33:50.047612311 +0000 UTC m=+5.706087312"
Sep 17 17:33:50.386037 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.385953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:50.386037 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.386015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:50.386267 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.386172    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.386267 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.386233    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.386215009 +0000 UTC m=+10.044689990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:50.386627 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.386504    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.386627 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.386557    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.386540798 +0000 UTC m=+10.045015796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:50.487387 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.486825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:50.487387 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.486963    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:50.487387 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.486984    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:50.487387 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.486996    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:50.487387 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.487052    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:33:54.4870349 +0000 UTC m=+10.145509884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:50.845620 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.845545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:50.845779 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.845666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:50.846039 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.846017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:50.846134 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.846112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:50.846215 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:50.846192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:50.846318 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:50.846300    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:52.845621 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:52.845584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:52.846101 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:52.845584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:52.846101 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:52.845832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:52.846101 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:52.845727    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:52.846101 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:52.845588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:52.846101 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:52.845922    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.421627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.421721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.421846    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.421902    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.421882986 +0000 UTC m=+18.080357969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.422570    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.422652 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.422621    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.422605211 +0000 UTC m=+18.081080203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:33:54.522651 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.522615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:54.522822 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.522795    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:33:54.522822 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.522822    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:33:54.522947 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.522836    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.522947 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.522899    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:02.522879933 +0000 UTC m=+18.181354910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:33:54.845361 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.845293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:54.845361 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.845315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:54.845606 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.845521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:54.846593 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.846560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:54.846676 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:54.846625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:54.846727 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:54.846703    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:56.845880 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:56.845845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:56.846357 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:56.845890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:56.846357 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:56.845844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:56.846357 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:56.845977    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:56.846357 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:56.846041    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:33:56.846357 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:56.846126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:58.845191 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:58.845156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:33:58.845758 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:58.845154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:33:58.845758 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:58.845298    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:33:58.845758 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:33:58.845154    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:33:58.845758 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:58.845372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:33:58.845758 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:33:58.845432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:00.844753 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:00.844709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:00.844753 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:00.844749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:00.845239 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:00.844710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:00.845239 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:00.844837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:00.845239 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:00.844956    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:00.845239 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:00.845025    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:02.476915 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.476883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:02.477406 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.476960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:02.477406 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.477043    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.477406 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.477110    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.477094893 +0000 UTC m=+34.135569872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:02.477406 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.477043    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.477406 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.477154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.477143122 +0000 UTC m=+34.135618100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:02.577229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.577194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:02.577422 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.577387    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:02.577488 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.577434    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:02.577488 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.577448    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.577591 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.577507    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:18.577487164 +0000 UTC m=+34.235962148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:02.845610 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.845537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:02.845767 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.845537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:02.845767 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.845657    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:02.845881 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.845790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:02.845881 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:02.845820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:02.845982 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:02.845905    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:04.845308 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:04.845229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:04.845740 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:04.845321    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:04.845740 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:04.845332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:04.845740 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:04.845440    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:04.845740 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:04.845461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:04.845740 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:04.845519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:06.068117 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.067649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vgkv9" event={"ID":"8296a988-6fdf-40e4-9b36-c2fa1b9d89e2","Type":"ContainerStarted","Data":"04c7c0342db6484ac04e0412afc7ff2b73879386c965bab4c92202d186d752af"}
Sep 17 17:34:06.069300 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.069268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7xgx" event={"ID":"263b9c4f-3fea-44fe-8f45-2da654828fca","Type":"ContainerStarted","Data":"51714171ef2e46b9478f335c4dcd9bfac7d0ca09fcd3be815c7f40f84f736ac3"}
Sep 17 17:34:06.070752 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.070727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5wf4z" event={"ID":"5a6db059-b74f-4db5-bca9-80090a1b1fae","Type":"ContainerStarted","Data":"aa8645aed7d29b008853fa3a3b4be1f08d558ffe8a5ff01db0e932cd0d16c84c"}
Sep 17 17:34:06.072450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.072426    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw" event={"ID":"c8449501-bfe5-45cf-81ae-35547447ae4f","Type":"ContainerStarted","Data":"05107a88017fef25077c9aa75ce2521d38b411101454837360473b7545956ed4"}
Sep 17 17:34:06.073973 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.073954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cfzd2" event={"ID":"060d1280-c356-48bc-8a89-4ddb085bf775","Type":"ContainerStarted","Data":"5494a4a3cfefcb7af728be44fdf07711d3e7e74025e23713904fa41b64663f5a"}
Sep 17 17:34:06.075615 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.075594    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="1bff79fa1057a42e29c8df24165429d9a244cd42d97dd30f0efcf027f969bd64" exitCode=0
Sep 17 17:34:06.075694 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.075657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"1bff79fa1057a42e29c8df24165429d9a244cd42d97dd30f0efcf027f969bd64"}
Sep 17 17:34:06.077230 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.077190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dq6jm" event={"ID":"2fbabe68-32c9-4772-8316-ccc2d0c949fa","Type":"ContainerStarted","Data":"5b2be71adfe58923235b8227ce62ce55b91c25802aac68e310cfb7083d061d00"}
Sep 17 17:34:06.079290 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.079190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"19fbd4b40b1ad04e73e5a66883b59eb713e3ad0f13324891078baea9f46ee5e7"}
Sep 17 17:34:06.079290 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.079257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"f28772e7c015f6f5a021d396c5472f862df7e37f5a8501f35430ffc5fdd347e3"}
Sep 17 17:34:06.098043 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.097982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vgkv9" podStartSLOduration=3.446605117 podStartE2EDuration="21.097966489s" totalImagesPullingTime="17.651361372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.083268009 +0000 UTC m=+21.741743010" watchObservedRunningTime="2026-09-17 17:34:06.097966489 +0000 UTC m=+21.756441488"
Sep 17 17:34:06.098229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.098204    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cfzd2" podStartSLOduration=3.431934908 podStartE2EDuration="21.098197257s" totalImagesPullingTime="17.666262349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.097336993 +0000 UTC m=+21.755811995" watchObservedRunningTime="2026-09-17 17:34:06.098197257 +0000 UTC m=+21.756672257"
Sep 17 17:34:06.136640 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.136590    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5wf4z" podStartSLOduration=3.47745999 podStartE2EDuration="21.136573037s" totalImagesPullingTime="17.659113047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.136296266 +0000 UTC m=+21.794771266" watchObservedRunningTime="2026-09-17 17:34:06.136573037 +0000 UTC m=+21.795048037"
Sep 17 17:34:06.150331 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.150288    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7xgx" podStartSLOduration=3.482259626 podStartE2EDuration="21.150272681s" totalImagesPullingTime="17.668013055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.149919531 +0000 UTC m=+21.808394530" watchObservedRunningTime="2026-09-17 17:34:06.150272681 +0000 UTC m=+21.808747681"
Sep 17 17:34:06.845346 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.845312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:06.845346 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.845334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:06.845562 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:06.845459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:06.845562 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.845496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:06.845657 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:06.845587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:06.845711 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:06.845656    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:06.962992 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:06.962960    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:07.084001 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.083969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"90b471fd598039af352cbad4c0fb5d36c244186bf71fe90bd21039a7e16ff61a"}
Sep 17 17:34:07.084683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.084049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"5c36f4763b721306e072f5f7c7e084cda7520704c75a4b8fcf6030cff65ea81e"}
Sep 17 17:34:07.084683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.084093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"5387bb997d151bf8062f41bac298b0b02f3ee07a520a2bb7130148435df16181"}
Sep 17 17:34:07.084683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.084125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"0ea04235765934244546d181ba26ff6eb2919071402b2c2476554106667549a8"}
Sep 17 17:34:07.085848 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.085826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kjp9" event={"ID":"6e1ba5e5-8de3-429e-84ae-960f72c59804","Type":"ContainerStarted","Data":"2533942b47fee41833ff9e957e4fecddc9d33ff3247934ff6581150293ab5285"}
Sep 17 17:34:07.087525 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.087502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw" event={"ID":"c8449501-bfe5-45cf-81ae-35547447ae4f","Type":"ContainerStarted","Data":"bc6a06dfca16278ef4229f756d937ace5e321cb465e87ec7db55519a4e469957"}
Sep 17 17:34:07.104805 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.104763    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kjp9" podStartSLOduration=5.429311098 podStartE2EDuration="23.104752399s" totalImagesPullingTime="17.675441301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:07.104692358 +0000 UTC m=+22.763167359" watchObservedRunningTime="2026-09-17 17:34:07.104752399 +0000 UTC m=+22.763227399"
Sep 17 17:34:07.105257 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.105228    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dq6jm" podStartSLOduration=4.102019732 podStartE2EDuration="22.105220725s" totalImagesPullingTime="18.003200993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:06.180216844 +0000 UTC m=+21.838691844" watchObservedRunningTime="2026-09-17 17:34:07.105220725 +0000 UTC m=+22.763695703"
Sep 17 17:34:07.821316 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.821220    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:06.962977737Z","UUID":"b956cf51-ae72-4936-b9da-0d0eae8df7b8","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:07.824134 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.824109    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:07.824281 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:07.824150    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:08.092227 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.092150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw" event={"ID":"c8449501-bfe5-45cf-81ae-35547447ae4f","Type":"ContainerStarted","Data":"93135ae8b4ac3ffe3a30d35f394ac03a2b47a8ad225ef1ed268ceae822f72551"}
Sep 17 17:34:08.114338 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.114289    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdwjw" podStartSLOduration=3.546730047 podStartE2EDuration="23.114267998s" totalImagesPullingTime="19.567537951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:08.113327824 +0000 UTC m=+23.771802824" watchObservedRunningTime="2026-09-17 17:34:08.114267998 +0000 UTC m=+23.772742999"
Sep 17 17:34:08.419495 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.419426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:34:08.420038 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.420023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:34:08.845482 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.845361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:08.845482 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.845366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:08.845682 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:08.845476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:08.845682 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:08.845366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:08.845682 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:08.845560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:08.845682 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:08.845668    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:09.097509 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:09.097285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"021297baf5283ed20141bba35cfa703a7756440711ebfaf105742ef2731a28e6"}
Sep 17 17:34:09.100362 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:09.099025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:34:09.100362 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:09.099176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vgkv9"
Sep 17 17:34:10.845177 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:10.845143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:10.845598 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:10.845145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:10.845598 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:10.845294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:10.845598 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:10.845306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:10.845598 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:10.845380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:10.845598 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:10.845509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:11.103592 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.103562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerStarted","Data":"516bee38ba9b75f80bc55270a64d1534190c6d85c51756a0ddacc9ff738dcc1f"}
Sep 17 17:34:11.107815 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.107786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" event={"ID":"1899b110-afd4-4abf-b9bf-fdafc8354d58","Type":"ContainerStarted","Data":"cd08ba26682d4035691631de7df1c1345cac203c925e22bf42fee06c33bdb046"}
Sep 17 17:34:11.108471 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.108081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:11.108471 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.108151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:11.123489 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.123274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:11.125201 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.125176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:11.147858 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.147809    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt" podStartSLOduration=8.442604986 podStartE2EDuration="26.147791393s" totalImagesPullingTime="17.705186407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:11.147698579 +0000 UTC m=+26.806173603" watchObservedRunningTime="2026-09-17 17:34:11.147791393 +0000 UTC m=+26.806266394"
Sep 17 17:34:11.273905 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:11.273882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:12.111609 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:12.111572    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="516bee38ba9b75f80bc55270a64d1534190c6d85c51756a0ddacc9ff738dcc1f" exitCode=0
Sep 17 17:34:12.111966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:12.111637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"516bee38ba9b75f80bc55270a64d1534190c6d85c51756a0ddacc9ff738dcc1f"}
Sep 17 17:34:12.845502 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:12.845466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:12.845502 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:12.845501    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:12.845720 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:12.845465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:12.845720 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:12.845592    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:12.845720 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:12.845672    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:12.845720 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:12.845713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:13.115914 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.115851    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="68b04e6ec0535962dd39265bb90ef9f530f4b4e76f99b6d0acfcf79261566c3c" exitCode=0
Sep 17 17:34:13.116288 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.115931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"68b04e6ec0535962dd39265bb90ef9f530f4b4e76f99b6d0acfcf79261566c3c"}
Sep 17 17:34:13.332094 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.332064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nplrw"]
Sep 17 17:34:13.332320 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.332156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:13.332320 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:13.332229    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:13.338834 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.338810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v424b"]
Sep 17 17:34:13.338957 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.338895    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:13.339077 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:13.339046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:13.339450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.339429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvfrz"]
Sep 17 17:34:13.339535 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:13.339521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:13.339642 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:13.339623    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:14.123251 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:14.123181    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="e546632d68330f8bac470c37fc2ae0212f8e64369cb9ab38e014eb82477cb11b" exitCode=0
Sep 17 17:34:14.123251 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:14.123240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"e546632d68330f8bac470c37fc2ae0212f8e64369cb9ab38e014eb82477cb11b"}
Sep 17 17:34:14.846011 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:14.845982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:14.846011 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:14.846002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:14.846212 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:14.846101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:14.846212 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:14.846158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:14.846316 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:14.846249    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:14.846369 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:14.846314    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:16.844983 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:16.844950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:16.845539 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:16.844955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:16.845539 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:16.845078    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v424b" podUID="3df4d312-893e-47bb-8437-c488710ef06d"
Sep 17 17:34:16.845539 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:16.845093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:16.845539 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:16.845174    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gvfrz" podUID="9d866f17-0e45-46eb-96c2-3638d706960c"
Sep 17 17:34:16.845539 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:16.845261    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nplrw" podUID="acebde19-4743-4eaf-9037-c00785c8e0b1"
Sep 17 17:34:18.499604 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.499573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:18.499954 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.499646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:18.499954 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.499738    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.499954 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.499746    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.499954 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.499796    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.499780087 +0000 UTC m=+66.158255068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:18.499954 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.499812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret podName:acebde19-4743-4eaf-9037-c00785c8e0b1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.499806284 +0000 UTC m=+66.158281263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret") pod "global-pull-secret-syncer-nplrw" (UID: "acebde19-4743-4eaf-9037-c00785c8e0b1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:18.600967 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.600932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:18.601125 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.601100    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:18.601192 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.601136    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:18.601192 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.601149    2577 projected.go:196] Error preparing data for projected volume kube-api-access-jsvkc for pod openshift-network-diagnostics/network-check-target-v424b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.601295 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:18.601208    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc podName:3df4d312-893e-47bb-8437-c488710ef06d nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.601193794 +0000 UTC m=+66.259668787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jsvkc" (UniqueName: "kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc") pod "network-check-target-v424b" (UID: "3df4d312-893e-47bb-8437-c488710ef06d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:18.688502 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.688444    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-61.ec2.internal" event="NodeReady"
Sep 17 17:34:18.688602 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.688547    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:18.730006 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.729985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:34:18.744137 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.744117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.744265 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.744248    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mh8gs"]
Sep 17 17:34:18.746318 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.746290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:34:18.746435 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.746291    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:34:18.746435 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.746375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-256jv\""
Sep 17 17:34:18.746647 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.746633    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:34:18.752989 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.752707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:34:18.760317 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.760293    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xpqnj"]
Sep 17 17:34:18.760438 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.760422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:18.762581 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.762421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:18.762581 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.762434    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:34:18.762581 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.762455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:18.788744 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.788724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:34:18.788825 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.788748    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mh8gs"]
Sep 17 17:34:18.788825 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.788764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpqnj"]
Sep 17 17:34:18.788900 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.788827    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:18.790629 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.790612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:18.790772 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.790732    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5qv6c\""
Sep 17 17:34:18.790904 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.790802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:18.791000 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.790978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:18.844735 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.844716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:18.844826 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.844733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:18.844884 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.844865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:18.849135 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:18.849286 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:18.849367 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:34:18.849450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8blr\""
Sep 17 17:34:18.849450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:18.849923 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.849902    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:18.903577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903666 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0526a7e-2c2b-4892-ae86-883f3253ae5a-tmp-dir\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:18.903666 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96nsl\" (UniqueName: \"kubernetes.io/projected/e0526a7e-2c2b-4892-ae86-883f3253ae5a-kube-api-access-96nsl\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:18.903736 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903736 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903736 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhlg\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.903844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:18.903844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f41ff3e-865a-44f8-88ae-efa0b62504f4-cert\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:18.904021 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxwf\" (UniqueName: \"kubernetes.io/projected/9f41ff3e-865a-44f8-88ae-efa0b62504f4-kube-api-access-lqxwf\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:18.904021 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0526a7e-2c2b-4892-ae86-883f3253ae5a-config-volume\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:18.904021 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.903900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:18.939060 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:18.938991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgkv9_8296a988-6fdf-40e4-9b36-c2fa1b9d89e2/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:19.004905 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.004881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96nsl\" (UniqueName: \"kubernetes.io/projected/e0526a7e-2c2b-4892-ae86-883f3253ae5a-kube-api-access-96nsl\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.005005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.004922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.004943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.004963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005005 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.004984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhlg\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.005041    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.005057    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.005102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:19.505086869 +0000 UTC m=+35.163561865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.005204 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f41ff3e-865a-44f8-88ae-efa0b62504f4-cert\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.005320    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005329    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxwf\" (UniqueName: \"kubernetes.io/projected/9f41ff3e-865a-44f8-88ae-efa0b62504f4-kube-api-access-lqxwf\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.005383    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:19.505370528 +0000 UTC m=+35.163845511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0526a7e-2c2b-4892-ae86-883f3253ae5a-config-volume\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.005577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0526a7e-2c2b-4892-ae86-883f3253ae5a-tmp-dir\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.005936 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.005888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0526a7e-2c2b-4892-ae86-883f3253ae5a-tmp-dir\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.006072 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.006054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.006116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.006075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.006463 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.006443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e0526a7e-2c2b-4892-ae86-883f3253ae5a-config-volume\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.009206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.009185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.009306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.009229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.009352 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.009334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f41ff3e-865a-44f8-88ae-efa0b62504f4-cert\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:19.014172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.014150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxwf\" (UniqueName: \"kubernetes.io/projected/9f41ff3e-865a-44f8-88ae-efa0b62504f4-kube-api-access-lqxwf\") pod \"ingress-canary-xpqnj\" (UID: \"9f41ff3e-865a-44f8-88ae-efa0b62504f4\") " pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:19.014701 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.014677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhlg\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.015286 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.015264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96nsl\" (UniqueName: \"kubernetes.io/projected/e0526a7e-2c2b-4892-ae86-883f3253ae5a-kube-api-access-96nsl\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.015706 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.015685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.050798 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.050771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vgkv9_8296a988-6fdf-40e4-9b36-c2fa1b9d89e2/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:19.096870 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.096848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpqnj"
Sep 17 17:34:19.436723 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.436693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpqnj"]
Sep 17 17:34:19.440766 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:34:19.440736    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f41ff3e_865a_44f8_88ae_efa0b62504f4.slice/crio-1297b67228e4c417597fec3b61db80f0c40254a761a45d2125ce0a957eda6be1 WatchSource:0}: Error finding container 1297b67228e4c417597fec3b61db80f0c40254a761a45d2125ce0a957eda6be1: Status 404 returned error can't find the container with id 1297b67228e4c417597fec3b61db80f0c40254a761a45d2125ce0a957eda6be1
Sep 17 17:34:19.509074 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.509050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:19.509331 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:19.509118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:19.509331 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.509223    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.509331 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.509280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.509265983 +0000 UTC m=+36.167740964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:19.509331 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.509234    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:19.509331 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.509323    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:19.509521 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:19.509362    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:20.509351525 +0000 UTC m=+36.167826516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:20.140098 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:20.139840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpqnj" event={"ID":"9f41ff3e-865a-44f8-88ae-efa0b62504f4","Type":"ContainerStarted","Data":"1297b67228e4c417597fec3b61db80f0c40254a761a45d2125ce0a957eda6be1"}
Sep 17 17:34:20.143251 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:20.143223    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="80a98d5218eeeb64c882c5c858289f06dc9d37a25b47f740ddc4a2c562af36d5" exitCode=0
Sep 17 17:34:20.143383 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:20.143262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"80a98d5218eeeb64c882c5c858289f06dc9d37a25b47f740ddc4a2c562af36d5"}
Sep 17 17:34:20.517687 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:20.517657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:20.517707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:20.517801    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:20.517820    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:20.517866    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:22.517850345 +0000 UTC m=+38.176325328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:20.517809    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:20.518004 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:20.517919    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:22.517907742 +0000 UTC m=+38.176382735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:21.148374 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:21.148339    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c18a0f8-b17b-4da8-82f0-e4282e527a70" containerID="2fb2045dbb9bba411647a1a315c3498035fe567f907aced40081ade05b8fede5" exitCode=0
Sep 17 17:34:21.148562 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:21.148421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerDied","Data":"2fb2045dbb9bba411647a1a315c3498035fe567f907aced40081ade05b8fede5"}
Sep 17 17:34:22.154621 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:22.154585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" event={"ID":"9c18a0f8-b17b-4da8-82f0-e4282e527a70","Type":"ContainerStarted","Data":"34196dc4ed43b48e7cc4f7314e3052cd20b06dd090752c202ebe689f3dfa9d17"}
Sep 17 17:34:22.180238 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:22.180195    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lg9m4" podStartSLOduration=8.105093584 podStartE2EDuration="37.180181585s" totalImagesPullingTime="29.075088001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:22.179271714 +0000 UTC m=+37.837746713" watchObservedRunningTime="2026-09-17 17:34:22.180181585 +0000 UTC m=+37.838656585"
Sep 17 17:34:22.534599 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:22.534520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:22.534599 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:22.534576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:22.534821 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:22.534682    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:22.534821 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:22.534692    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:22.534821 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:22.534718    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:22.534821 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:22.534731    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.534718513 +0000 UTC m=+42.193193491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:22.534821 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:22.534784    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.534766682 +0000 UTC m=+42.193241671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:23.158410 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:23.158357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpqnj" event={"ID":"9f41ff3e-865a-44f8-88ae-efa0b62504f4","Type":"ContainerStarted","Data":"794329009c10973a61928c99b34894618489f25c6b9bd5685f3b37af49569d34"}
Sep 17 17:34:23.176079 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:23.176035    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xpqnj" podStartSLOduration=2.341178271 podStartE2EDuration="5.176021355s" totalImagesPullingTime="2.834843084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:23.174489993 +0000 UTC m=+38.832964994" watchObservedRunningTime="2026-09-17 17:34:23.176021355 +0000 UTC m=+38.834496355"
Sep 17 17:34:26.564503 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:26.564423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:26.564503 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:26.564492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:26.564866 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:26.564595    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:26.564866 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:26.564599    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.564866 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:26.564666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.5646505 +0000 UTC m=+50.223125477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:26.564866 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:26.564606    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:26.564866 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:26.564727    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:34.564716219 +0000 UTC m=+50.223191230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:34.621688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:34.621640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:34.621688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:34.621695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:34.622200 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:34.621798    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.622200 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:34.621803    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:34.622200 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:34.621829    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:34.622200 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:34.621859    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.621844349 +0000 UTC m=+66.280319326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:34.622200 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:34.621895    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:34:50.62187961 +0000 UTC m=+66.280354587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:40.080677 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.080637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"]
Sep 17 17:34:40.082723 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.082699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.084858 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.084830    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.085141 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.085100    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:34:40.085654 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.085634    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sbj7s\""
Sep 17 17:34:40.085760 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.085653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:40.108504 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.108482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"]
Sep 17 17:34:40.157900 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.157873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49f9\" (UniqueName: \"kubernetes.io/projected/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-kube-api-access-l49f9\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.158000 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.157910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.259099 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.259074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.259213 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.259154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49f9\" (UniqueName: \"kubernetes.io/projected/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-kube-api-access-l49f9\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.259273 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:40.259232    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:40.259325 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:40.259312    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:40.759290577 +0000 UTC m=+56.417765566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:40.271717 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.271696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49f9\" (UniqueName: \"kubernetes.io/projected/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-kube-api-access-l49f9\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.762634 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.762603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:40.762771 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:40.762756    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:40.762856 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:40.762843    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.762828356 +0000 UTC m=+57.421303335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:40.990577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.990543    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cl69d"]
Sep 17 17:34:40.994424 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.994386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:40.996412 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.996372    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:34:40.996412 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.996405    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:40.996575 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.996553    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:34:40.997116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.997094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:40.997226 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:40.997102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-w7gbq\""
Sep 17 17:34:41.000562 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.000543    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:34:41.006566 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.006544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cl69d"]
Sep 17 17:34:41.065068 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.065017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-config\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.065068 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.065042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx45\" (UniqueName: \"kubernetes.io/projected/517ec90c-e584-4cd8-a7f0-f8a1e7207267-kube-api-access-ktx45\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.065177 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.065083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/517ec90c-e584-4cd8-a7f0-f8a1e7207267-serving-cert\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.065177 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.065126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-trusted-ca\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.166357 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.166330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-trusted-ca\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.166750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.166420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-config\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.166750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.166438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx45\" (UniqueName: \"kubernetes.io/projected/517ec90c-e584-4cd8-a7f0-f8a1e7207267-kube-api-access-ktx45\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.166750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.166489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/517ec90c-e584-4cd8-a7f0-f8a1e7207267-serving-cert\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.167135 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.167111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-config\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.167208 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.167192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/517ec90c-e584-4cd8-a7f0-f8a1e7207267-trusted-ca\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.168455 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.168440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/517ec90c-e584-4cd8-a7f0-f8a1e7207267-serving-cert\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.178891 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.178867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx45\" (UniqueName: \"kubernetes.io/projected/517ec90c-e584-4cd8-a7f0-f8a1e7207267-kube-api-access-ktx45\") pod \"console-operator-fd4bb46c-cl69d\" (UID: \"517ec90c-e584-4cd8-a7f0-f8a1e7207267\") " pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.303778 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.303755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:41.585546 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.585510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cl69d"]
Sep 17 17:34:41.589937 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:34:41.589911    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517ec90c_e584_4cd8_a7f0_f8a1e7207267.slice/crio-824b1f162402ec01e936c208455af3b3da7dd63cd58944e595b5d43af7e4d7dc WatchSource:0}: Error finding container 824b1f162402ec01e936c208455af3b3da7dd63cd58944e595b5d43af7e4d7dc: Status 404 returned error can't find the container with id 824b1f162402ec01e936c208455af3b3da7dd63cd58944e595b5d43af7e4d7dc
Sep 17 17:34:41.771106 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:41.771085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:41.771214 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:41.771192    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:41.771254 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:41.771236    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:43.771224263 +0000 UTC m=+59.429699241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:42.203729 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:42.203691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" event={"ID":"517ec90c-e584-4cd8-a7f0-f8a1e7207267","Type":"ContainerStarted","Data":"824b1f162402ec01e936c208455af3b3da7dd63cd58944e595b5d43af7e4d7dc"}
Sep 17 17:34:43.127576 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.127540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qvqqt"
Sep 17 17:34:43.558557 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.558225    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qg5lf"]
Sep 17 17:34:43.560508 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.560491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.562671 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.562642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:34:43.562779 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.562698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:34:43.562779 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.562761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:34:43.562932 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.562918    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:34:43.562977 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.562951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:34:43.579097 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.579077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qg5lf"]
Sep 17 17:34:43.587150 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.587126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.587243 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.587153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zs5b\" (UniqueName: \"kubernetes.io/projected/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-api-access-5zs5b\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.587243 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.587188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-crio-socket\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.587243 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.587226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.587354 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.587324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-data-volume\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688108 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-crio-socket\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688142    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-data-volume\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688352 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.688209    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.688352 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.688300    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.188277985 +0000 UTC m=+59.846752965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:43.688352 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zs5b\" (UniqueName: \"kubernetes.io/projected/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-api-access-5zs5b\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688352 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-crio-socket\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.688352 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.688334    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.688559 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.688425    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:44.188412326 +0000 UTC m=+59.846887307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:43.688559 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.688496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-data-volume\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.703963 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.703939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zs5b\" (UniqueName: \"kubernetes.io/projected/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-api-access-5zs5b\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:43.789279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:43.789240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:43.789440 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.789375    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:43.789507 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:43.789453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.789438771 +0000 UTC m=+63.447913781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:44.192382 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.192343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:44.192603 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:44.192472    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.192603 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.192489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:44.192603 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:44.192540    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.192524447 +0000 UTC m=+60.850999487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:44.192764 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:44.192620    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:44.192764 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:44.192698    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:45.192679585 +0000 UTC m=+60.851154596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:44.210135 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.210113    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/0.log"
Sep 17 17:34:44.210275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.210149    2577 generic.go:356] "Generic (PLEG): container finished" podID="517ec90c-e584-4cd8-a7f0-f8a1e7207267" containerID="b0dcdd301c605dddbd96f198a7c794a9fa32e83de166c3e9b57698f5296ae30b" exitCode=255
Sep 17 17:34:44.210275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.210199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" event={"ID":"517ec90c-e584-4cd8-a7f0-f8a1e7207267","Type":"ContainerDied","Data":"b0dcdd301c605dddbd96f198a7c794a9fa32e83de166c3e9b57698f5296ae30b"}
Sep 17 17:34:44.210497 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:44.210479    2577 scope.go:117] "RemoveContainer" containerID="b0dcdd301c605dddbd96f198a7c794a9fa32e83de166c3e9b57698f5296ae30b"
Sep 17 17:34:45.199044 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.199005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:45.199044 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.199061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:45.199551 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:45.199138    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.199551 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:45.199155    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.199551 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:45.199195    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.199178617 +0000 UTC m=+62.857653597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:45.199551 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:45.199208    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:47.199202341 +0000 UTC m=+62.857677318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:45.213824 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.213801    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/1.log"
Sep 17 17:34:45.214199 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.214184    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/0.log"
Sep 17 17:34:45.214250 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.214217    2577 generic.go:356] "Generic (PLEG): container finished" podID="517ec90c-e584-4cd8-a7f0-f8a1e7207267" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f" exitCode=255
Sep 17 17:34:45.214317 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.214302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" event={"ID":"517ec90c-e584-4cd8-a7f0-f8a1e7207267","Type":"ContainerDied","Data":"c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"}
Sep 17 17:34:45.214364 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.214353    2577 scope.go:117] "RemoveContainer" containerID="b0dcdd301c605dddbd96f198a7c794a9fa32e83de166c3e9b57698f5296ae30b"
Sep 17 17:34:45.214547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.214529    2577 scope.go:117] "RemoveContainer" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"
Sep 17 17:34:45.214691 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:45.214672    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:34:45.498929 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:45.498871    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7xgx_263b9c4f-3fea-44fe-8f45-2da654828fca/dns-node-resolver/0.log"
Sep 17 17:34:46.218190 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:46.218163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/1.log"
Sep 17 17:34:46.218569 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:46.218475    2577 scope.go:117] "RemoveContainer" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"
Sep 17 17:34:46.218634 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:46.218614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:34:46.303875 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:46.303854    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfzd2_060d1280-c356-48bc-8a89-4ddb085bf775/node-ca/0.log"
Sep 17 17:34:47.214754 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:47.214719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:47.214958 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:47.214771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:47.214958 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.214905    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.214958 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.214938    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.215130 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.214989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.2149673 +0000 UTC m=+66.873442334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:47.215130 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.215012    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:51.215002208 +0000 UTC m=+66.873477193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:47.501954 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:47.501880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpqnj_9f41ff3e-865a-44f8-88ae-efa0b62504f4/serve-healthcheck-canary/0.log"
Sep 17 17:34:47.819957 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:47.819871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:47.820102 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.819988    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:47.820102 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:47.820045    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.820032027 +0000 UTC m=+71.478507005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:50.539811 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.539775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:50.540214 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.539857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:34:50.542211 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.542193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:50.542211 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.542205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:50.550099 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.550076    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:34:50.550154 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.550138    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs podName:9d866f17-0e45-46eb-96c2-3638d706960c nodeName:}" failed. No retries permitted until 2026-09-17 17:35:54.550121309 +0000 UTC m=+130.208596292 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs") pod "network-metrics-daemon-gvfrz" (UID: "9d866f17-0e45-46eb-96c2-3638d706960c") : secret "metrics-daemon-secret" not found
Sep 17 17:34:50.552155 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.552123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/acebde19-4743-4eaf-9037-c00785c8e0b1-original-pull-secret\") pod \"global-pull-secret-syncer-nplrw\" (UID: \"acebde19-4743-4eaf-9037-c00785c8e0b1\") " pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:50.640910 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.640867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:34:50.641062 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.640940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:50.641062 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.640967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:34:50.641062 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.641013    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:34:50.641062 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.641033    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d984ffc-25jvh: secret "image-registry-tls" not found
Sep 17 17:34:50.641255 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.641087    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls podName:dbf374ae-adb7-4b3a-945c-abe14dd44caf nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.641069199 +0000 UTC m=+98.299544190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls") pod "image-registry-846d984ffc-25jvh" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf") : secret "image-registry-tls" not found
Sep 17 17:34:50.641255 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.641125    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.641255 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:50.641182    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls podName:e0526a7e-2c2b-4892-ae86-883f3253ae5a nodeName:}" failed. No retries permitted until 2026-09-17 17:35:22.641166397 +0000 UTC m=+98.299641390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls") pod "dns-default-mh8gs" (UID: "e0526a7e-2c2b-4892-ae86-883f3253ae5a") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:50.643435 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.643418    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:50.653329 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.653313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:50.663785 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.663768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvkc\" (UniqueName: \"kubernetes.io/projected/3df4d312-893e-47bb-8437-c488710ef06d-kube-api-access-jsvkc\") pod \"network-check-target-v424b\" (UID: \"3df4d312-893e-47bb-8437-c488710ef06d\") " pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:50.667529 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.667512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nplrw"
Sep 17 17:34:50.935407 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.935359    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nplrw"]
Sep 17 17:34:50.938404 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:34:50.938361    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacebde19_4743_4eaf_9037_c00785c8e0b1.slice/crio-a7a7acb44f82a21cd633f3f695a76170a6da0c7c2f9dd26aea3bb7f3fee08baa WatchSource:0}: Error finding container a7a7acb44f82a21cd633f3f695a76170a6da0c7c2f9dd26aea3bb7f3fee08baa: Status 404 returned error can't find the container with id a7a7acb44f82a21cd633f3f695a76170a6da0c7c2f9dd26aea3bb7f3fee08baa
Sep 17 17:34:50.959194 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.959172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z8blr\""
Sep 17 17:34:50.967944 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:50.967924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:51.230244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.230171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nplrw" event={"ID":"acebde19-4743-4eaf-9037-c00785c8e0b1","Type":"ContainerStarted","Data":"a7a7acb44f82a21cd633f3f695a76170a6da0c7c2f9dd26aea3bb7f3fee08baa"}
Sep 17 17:34:51.233088 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.233064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v424b"]
Sep 17 17:34:51.235739 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:34:51.235719    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df4d312_893e_47bb_8437_c488710ef06d.slice/crio-122643ca7e13afb9af0e0c2fb5766c003c2efceaeb35e0a33f6fdfa77e69eed1 WatchSource:0}: Error finding container 122643ca7e13afb9af0e0c2fb5766c003c2efceaeb35e0a33f6fdfa77e69eed1: Status 404 returned error can't find the container with id 122643ca7e13afb9af0e0c2fb5766c003c2efceaeb35e0a33f6fdfa77e69eed1
Sep 17 17:34:51.245898 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.245880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:51.245957 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.245922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:51.245997 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.245985    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.246044 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.246011    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:51.246044 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.246031    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.246019191 +0000 UTC m=+74.904494169 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : configmap "kube-rbac-proxy" not found
Sep 17 17:34:51.246124 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.246052    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.246040317 +0000 UTC m=+74.904515299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:51.304696 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.304678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:51.304777 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.304716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:34:51.304996 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.304985    2577 scope.go:117] "RemoveContainer" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"
Sep 17 17:34:51.305117 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.305102    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:34:51.340619 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.340594    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"]
Sep 17 17:34:51.342776 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.342762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:51.344948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.344924    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:34:51.345046 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.344927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hv2sf\""
Sep 17 17:34:51.345046 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.344965    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:34:51.359775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.359752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"]
Sep 17 17:34:51.447352 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.447324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc988e16-a3cd-49c4-96bf-5ecc628daec7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:51.447455 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.447360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:51.548493 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.548424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc988e16-a3cd-49c4-96bf-5ecc628daec7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:51.548493 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.548457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:51.548920 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.548589    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.548920 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:51.548658    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert podName:dc988e16-a3cd-49c4-96bf-5ecc628daec7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:52.048639418 +0000 UTC m=+67.707114411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-h5lsq" (UID: "dc988e16-a3cd-49c4-96bf-5ecc628daec7") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:51.549120 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:51.549102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc988e16-a3cd-49c4-96bf-5ecc628daec7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:52.052693 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:52.052656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:52.052865 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:52.052816    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:52.052928 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:52.052886    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert podName:dc988e16-a3cd-49c4-96bf-5ecc628daec7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.05287142 +0000 UTC m=+68.711346398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-h5lsq" (UID: "dc988e16-a3cd-49c4-96bf-5ecc628daec7") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:52.236857 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:52.236775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v424b" event={"ID":"3df4d312-893e-47bb-8437-c488710ef06d","Type":"ContainerStarted","Data":"122643ca7e13afb9af0e0c2fb5766c003c2efceaeb35e0a33f6fdfa77e69eed1"}
Sep 17 17:34:53.060163 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:53.060125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:53.060616 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:53.060305    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:53.060616 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:53.060382    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert podName:dc988e16-a3cd-49c4-96bf-5ecc628daec7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:55.060362161 +0000 UTC m=+70.718837149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-h5lsq" (UID: "dc988e16-a3cd-49c4-96bf-5ecc628daec7") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:54.615230 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.615189    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-78xcj"]
Sep 17 17:34:54.617600 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.617576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.619891 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.619854    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:34:54.619985 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.619896    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:34:54.620576 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.620554    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:34:54.620933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.620770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:34:54.620933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.620780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dwjp7\""
Sep 17 17:34:54.634787 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.634688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-78xcj"]
Sep 17 17:34:54.673590 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.673561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjhf5\" (UniqueName: \"kubernetes.io/projected/ed27bd85-c953-405f-8c9b-aa4c9a34629e-kube-api-access-vjhf5\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.673748 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.673645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-key\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.673748 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.673697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-cabundle\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.774905 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.774871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-cabundle\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.775095 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.774965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjhf5\" (UniqueName: \"kubernetes.io/projected/ed27bd85-c953-405f-8c9b-aa4c9a34629e-kube-api-access-vjhf5\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.775095 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.775052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-key\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.775668 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.775640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-cabundle\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.777668 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.777645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed27bd85-c953-405f-8c9b-aa4c9a34629e-signing-key\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.784336 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.784315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjhf5\" (UniqueName: \"kubernetes.io/projected/ed27bd85-c953-405f-8c9b-aa4c9a34629e-kube-api-access-vjhf5\") pod \"service-ca-55f6fc7b86-78xcj\" (UID: \"ed27bd85-c953-405f-8c9b-aa4c9a34629e\") " pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:54.928335 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:54.928258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj"
Sep 17 17:34:55.076763 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:55.076720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:55.076927 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:55.076863    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:55.076993 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:55.076939    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert podName:dc988e16-a3cd-49c4-96bf-5ecc628daec7 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.076919021 +0000 UTC m=+74.735394006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-h5lsq" (UID: "dc988e16-a3cd-49c4-96bf-5ecc628daec7") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:55.777146 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:55.777120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-78xcj"]
Sep 17 17:34:55.779351 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:34:55.779322    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded27bd85_c953_405f_8c9b_aa4c9a34629e.slice/crio-0987bd3d71c5a1725f3995bfeaad4e09238401b9860feb8fc9a7e7af69a90b91 WatchSource:0}: Error finding container 0987bd3d71c5a1725f3995bfeaad4e09238401b9860feb8fc9a7e7af69a90b91: Status 404 returned error can't find the container with id 0987bd3d71c5a1725f3995bfeaad4e09238401b9860feb8fc9a7e7af69a90b91
Sep 17 17:34:55.885210 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:55.885175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:34:55.885472 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:55.885341    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:34:55.885472 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:55.885425    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls podName:5a1ebdd5-88c7-44b4-a347-99ce4daa1c43 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.885408785 +0000 UTC m=+87.543883783 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-69l6r" (UID: "5a1ebdd5-88c7-44b4-a347-99ce4daa1c43") : secret "samples-operator-tls" not found
Sep 17 17:34:56.250330 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.250292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nplrw" event={"ID":"acebde19-4743-4eaf-9037-c00785c8e0b1","Type":"ContainerStarted","Data":"eaa9e1a63cec218bbda6ce8b282ffb46400921ad78a3a93ff4101e42efc4d46f"}
Sep 17 17:34:56.251731 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.251682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj" event={"ID":"ed27bd85-c953-405f-8c9b-aa4c9a34629e","Type":"ContainerStarted","Data":"0987bd3d71c5a1725f3995bfeaad4e09238401b9860feb8fc9a7e7af69a90b91"}
Sep 17 17:34:56.253255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.253223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v424b" event={"ID":"3df4d312-893e-47bb-8437-c488710ef06d","Type":"ContainerStarted","Data":"a7bc1e35f17c36ce021765140a162357b645b8ac961cb5a77e425c282d3d1b72"}
Sep 17 17:34:56.253387 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.253364    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:34:56.285802 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.285627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nplrw" podStartSLOduration=66.710666406 podStartE2EDuration="1m11.285608833s" totalImagesPullingTime="4.574942427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:56.267900137 +0000 UTC m=+71.926375138" watchObservedRunningTime="2026-09-17 17:34:56.285608833 +0000 UTC m=+71.944083834"
Sep 17 17:34:56.285983 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:56.285942    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v424b" podStartSLOduration=67.020778531 podStartE2EDuration="1m11.285933865s" totalImagesPullingTime="4.265155334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:56.285568194 +0000 UTC m=+71.944043195" watchObservedRunningTime="2026-09-17 17:34:56.285933865 +0000 UTC m=+71.944408880"
Sep 17 17:34:58.260207 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:58.260172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj" event={"ID":"ed27bd85-c953-405f-8c9b-aa4c9a34629e","Type":"ContainerStarted","Data":"63af333876679df482c218fe9205ea8fe3143ed280dc2bd840c53c234ae8b2e8"}
Sep 17 17:34:58.276207 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:58.276160    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-78xcj" podStartSLOduration=2.690787806 podStartE2EDuration="4.276146408s" totalImagesPullingTime="1.585358602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:58.275595423 +0000 UTC m=+73.934070418" watchObservedRunningTime="2026-09-17 17:34:58.276146408 +0000 UTC m=+73.934621407"
Sep 17 17:34:59.111455 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:59.111407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:34:59.111645 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:59.111500    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:34:59.111645 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:59.111576    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert podName:dc988e16-a3cd-49c4-96bf-5ecc628daec7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:07.111556252 +0000 UTC m=+82.770031236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-h5lsq" (UID: "dc988e16-a3cd-49c4-96bf-5ecc628daec7") : secret "networking-console-plugin-cert" not found
Sep 17 17:34:59.313279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:59.313247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:59.313718 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:59.313412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:34:59.313718 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:59.313532    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.313718 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:34:59.313599    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls podName:e3415f11-9bc0-4a2b-8ea4-1275704ef3ba nodeName:}" failed. No retries permitted until 2026-09-17 17:35:15.313580466 +0000 UTC m=+90.972055454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qg5lf" (UID: "e3415f11-9bc0-4a2b-8ea4-1275704ef3ba") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:34:59.313843 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:34:59.313823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:35:06.845494 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:06.845460    2577 scope.go:117] "RemoveContainer" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"
Sep 17 17:35:07.177494 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.177457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:35:07.179823 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.179790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc988e16-a3cd-49c4-96bf-5ecc628daec7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-h5lsq\" (UID: \"dc988e16-a3cd-49c4-96bf-5ecc628daec7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:35:07.251966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.251931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"
Sep 17 17:35:07.292018 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.291989    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:35:07.293599 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.293569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/1.log"
Sep 17 17:35:07.293721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.293615    2577 generic.go:356] "Generic (PLEG): container finished" podID="517ec90c-e584-4cd8-a7f0-f8a1e7207267" containerID="46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368" exitCode=255
Sep 17 17:35:07.293721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.293672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" event={"ID":"517ec90c-e584-4cd8-a7f0-f8a1e7207267","Type":"ContainerDied","Data":"46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368"}
Sep 17 17:35:07.293821 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.293750    2577 scope.go:117] "RemoveContainer" containerID="c94ecb8ebada21874aa5f00ebbf509fc83401fd7cf21a4fb110c31d0fd9cca4f"
Sep 17 17:35:07.294136 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.294118    2577 scope.go:117] "RemoveContainer" containerID="46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368"
Sep 17 17:35:07.294494 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:35:07.294469    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:35:07.528181 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:07.528051    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq"]
Sep 17 17:35:07.532500 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:07.532471    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc988e16_a3cd_49c4_96bf_5ecc628daec7.slice/crio-af08c076abd93ae4ca44cd0bb90d7d98be3f8512509d885f3d8ae8d4bb3613a3 WatchSource:0}: Error finding container af08c076abd93ae4ca44cd0bb90d7d98be3f8512509d885f3d8ae8d4bb3613a3: Status 404 returned error can't find the container with id af08c076abd93ae4ca44cd0bb90d7d98be3f8512509d885f3d8ae8d4bb3613a3
Sep 17 17:35:08.298058 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:08.298033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:35:08.299295 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:08.299268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq" event={"ID":"dc988e16-a3cd-49c4-96bf-5ecc628daec7","Type":"ContainerStarted","Data":"af08c076abd93ae4ca44cd0bb90d7d98be3f8512509d885f3d8ae8d4bb3613a3"}
Sep 17 17:35:09.303979 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:09.303944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq" event={"ID":"dc988e16-a3cd-49c4-96bf-5ecc628daec7","Type":"ContainerStarted","Data":"18d34ced6b3695a79c29c75ba083952d9126564b1d2048a2df9812fc98645f68"}
Sep 17 17:35:09.318914 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:09.318866    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-h5lsq" podStartSLOduration=17.222485174 podStartE2EDuration="18.318852765s" totalImagesPullingTime="1.096367591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:09.318477944 +0000 UTC m=+84.976952944" watchObservedRunningTime="2026-09-17 17:35:09.318852765 +0000 UTC m=+84.977327764"
Sep 17 17:35:11.304637 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:11.304600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:35:11.305095 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:11.304662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:35:11.305095 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:11.304916    2577 scope.go:117] "RemoveContainer" containerID="46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368"
Sep 17 17:35:11.305095 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:35:11.305050    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:35:11.912483 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:11.912448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:35:11.914835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:11.914804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1ebdd5-88c7-44b4-a347-99ce4daa1c43-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-69l6r\" (UID: \"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:35:12.194666 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:12.194601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sbj7s\""
Sep 17 17:35:12.203162 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:12.203148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"
Sep 17 17:35:12.478585 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:12.478513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r"]
Sep 17 17:35:13.320172 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:13.320136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r" event={"ID":"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43","Type":"ContainerStarted","Data":"a601156e4f42f605e0a6cc4e3c8fec8588fc8c397ce6974f90a02a8b6bb2d07e"}
Sep 17 17:35:14.324668 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:14.324631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r" event={"ID":"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43","Type":"ContainerStarted","Data":"4403d018d058d5edec2651fc9769ce9af358df8ce3623391dbc37dfb8944bef3"}
Sep 17 17:35:14.325088 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:14.324692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r" event={"ID":"5a1ebdd5-88c7-44b4-a347-99ce4daa1c43","Type":"ContainerStarted","Data":"d59e003e4f73c521e31b099f172188882465af8260ad251803a0dbe801198495"}
Sep 17 17:35:14.355794 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:14.355751    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-69l6r" podStartSLOduration=32.772745158 podStartE2EDuration="34.355737402s" totalImagesPullingTime="1.582992244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.354502039 +0000 UTC m=+90.012977039" watchObservedRunningTime="2026-09-17 17:35:14.355737402 +0000 UTC m=+90.014212401"
Sep 17 17:35:15.339322 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:15.339280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:35:15.341654 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:15.341630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3415f11-9bc0-4a2b-8ea4-1275704ef3ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qg5lf\" (UID: \"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba\") " pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:35:15.371718 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:15.371691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6zzl7\""
Sep 17 17:35:15.379477 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:15.379459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qg5lf"
Sep 17 17:35:15.654360 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:15.654333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qg5lf"]
Sep 17 17:35:15.657723 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:15.657691    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3415f11_9bc0_4a2b_8ea4_1275704ef3ba.slice/crio-2cde2ea25790b4b148bb67fbec79865f7915c157662402a8ba9b65ae38e701e7 WatchSource:0}: Error finding container 2cde2ea25790b4b148bb67fbec79865f7915c157662402a8ba9b65ae38e701e7: Status 404 returned error can't find the container with id 2cde2ea25790b4b148bb67fbec79865f7915c157662402a8ba9b65ae38e701e7
Sep 17 17:35:16.333594 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:16.333567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg5lf" event={"ID":"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba","Type":"ContainerStarted","Data":"e5d8f0d6269f73986da560bcf389aca36df2c3792a61d8c771489b3b51649d14"}
Sep 17 17:35:16.333696 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:16.333623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg5lf" event={"ID":"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba","Type":"ContainerStarted","Data":"2cde2ea25790b4b148bb67fbec79865f7915c157662402a8ba9b65ae38e701e7"}
Sep 17 17:35:17.338867 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:17.338830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg5lf" event={"ID":"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba","Type":"ContainerStarted","Data":"ba852de05875d5e28a4df1894b1c34bfd953660397159278224367ac08736b44"}
Sep 17 17:35:18.344617 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.344580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qg5lf" event={"ID":"e3415f11-9bc0-4a2b-8ea4-1275704ef3ba","Type":"ContainerStarted","Data":"4f00069dce643314fb23be3815e56526c2ac8bf610cbd254e8d5d910ecf0e91b"}
Sep 17 17:35:18.373182 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.373137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qg5lf" podStartSLOduration=33.645910879 podStartE2EDuration="35.373124406s" totalImagesPullingTime="1.727213527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:18.372434036 +0000 UTC m=+94.030909036" watchObservedRunningTime="2026-09-17 17:35:18.373124406 +0000 UTC m=+94.031599406"
Sep 17 17:35:18.512412 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.512368    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"]
Sep 17 17:35:18.515425 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.515410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.518185 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:35:18.518314 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:35:18.518314 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:35:18.518314 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:35:18.518314 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:35:18.518500 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:35:18.518500 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.518328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:35:18.536218 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.536196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"]
Sep 17 17:35:18.663183 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.663183 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.663377 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd9s6\" (UniqueName: \"kubernetes.io/projected/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-kube-api-access-jd9s6\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.663377 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.663377 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.663377 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.663330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764314 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd9s6\" (UniqueName: \"kubernetes.io/projected/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-kube-api-access-jd9s6\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.764462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.764451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.765056 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.765030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.766896 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.766868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-ca\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.767053 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.767034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.767327 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.767309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.767451 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.767433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-hub\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.785711 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.785689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd9s6\" (UniqueName: \"kubernetes.io/projected/8900e41d-6ee3-44f4-b146-c5bdc5da4ff6-kube-api-access-jd9s6\") pod \"cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z\" (UID: \"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:18.831691 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:18.831670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"
Sep 17 17:35:19.109310 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:19.109281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z"]
Sep 17 17:35:19.112405 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:19.112369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8900e41d_6ee3_44f4_b146_c5bdc5da4ff6.slice/crio-5fb25c87ff506eaf501c0840c5a08429b59a4d019db4e365af578b03efaedb4e WatchSource:0}: Error finding container 5fb25c87ff506eaf501c0840c5a08429b59a4d019db4e365af578b03efaedb4e: Status 404 returned error can't find the container with id 5fb25c87ff506eaf501c0840c5a08429b59a4d019db4e365af578b03efaedb4e
Sep 17 17:35:19.348738 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:19.348701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z" event={"ID":"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6","Type":"ContainerStarted","Data":"5fb25c87ff506eaf501c0840c5a08429b59a4d019db4e365af578b03efaedb4e"}
Sep 17 17:35:22.361031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.361000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z" event={"ID":"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6","Type":"ContainerStarted","Data":"b6910eb25a771e3735503931410a650e12474a661502e312db6a5f423596c35c"}
Sep 17 17:35:22.361326 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.361051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z" event={"ID":"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6","Type":"ContainerStarted","Data":"7ee284b2c0f67d40c5f542156c5d373aefd115a9dd94ac061004bdbe6a4f1e5a"}
Sep 17 17:35:22.700898 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.700863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:35:22.701041 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.700940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:35:22.703197 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.703169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0526a7e-2c2b-4892-ae86-883f3253ae5a-metrics-tls\") pod \"dns-default-mh8gs\" (UID: \"e0526a7e-2c2b-4892-ae86-883f3253ae5a\") " pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:35:22.703306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.703201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"image-registry-846d984ffc-25jvh\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") " pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:35:22.845244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.845207    2577 scope.go:117] "RemoveContainer" containerID="46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368"
Sep 17 17:35:22.845455 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:35:22.845431    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cl69d_openshift-console-operator(517ec90c-e584-4cd8-a7f0-f8a1e7207267)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podUID="517ec90c-e584-4cd8-a7f0-f8a1e7207267"
Sep 17 17:35:22.957229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.957154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-256jv\""
Sep 17 17:35:22.965678 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.965660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:35:22.970306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.970284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8dgbk\""
Sep 17 17:35:22.979268 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:22.979248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:35:23.245485 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.245410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:35:23.248633 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:23.248606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf374ae_adb7_4b3a_945c_abe14dd44caf.slice/crio-94ba768b465e1dae3e7d41295db4f2de5c3997404cfc3bb591920b22e38b5b22 WatchSource:0}: Error finding container 94ba768b465e1dae3e7d41295db4f2de5c3997404cfc3bb591920b22e38b5b22: Status 404 returned error can't find the container with id 94ba768b465e1dae3e7d41295db4f2de5c3997404cfc3bb591920b22e38b5b22
Sep 17 17:35:23.263795 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.263763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mh8gs"]
Sep 17 17:35:23.267217 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:23.267189    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0526a7e_2c2b_4892_ae86_883f3253ae5a.slice/crio-8bec6db9ac2d632df1e0bc3ed9a196435ae68fda0ae6d8dc4983c53c52afac8e WatchSource:0}: Error finding container 8bec6db9ac2d632df1e0bc3ed9a196435ae68fda0ae6d8dc4983c53c52afac8e: Status 404 returned error can't find the container with id 8bec6db9ac2d632df1e0bc3ed9a196435ae68fda0ae6d8dc4983c53c52afac8e
Sep 17 17:35:23.365541 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.365503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh8gs" event={"ID":"e0526a7e-2c2b-4892-ae86-883f3253ae5a","Type":"ContainerStarted","Data":"8bec6db9ac2d632df1e0bc3ed9a196435ae68fda0ae6d8dc4983c53c52afac8e"}
Sep 17 17:35:23.367157 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.367127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" event={"ID":"dbf374ae-adb7-4b3a-945c-abe14dd44caf","Type":"ContainerStarted","Data":"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"}
Sep 17 17:35:23.367289 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.367175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" event={"ID":"dbf374ae-adb7-4b3a-945c-abe14dd44caf","Type":"ContainerStarted","Data":"94ba768b465e1dae3e7d41295db4f2de5c3997404cfc3bb591920b22e38b5b22"}
Sep 17 17:35:23.367289 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.367204    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:35:23.369348 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.369320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z" event={"ID":"8900e41d-6ee3-44f4-b146-c5bdc5da4ff6","Type":"ContainerStarted","Data":"117f5b84920944bd481c9418678e7ccfad4783626b1e0379b530df80dd0e852d"}
Sep 17 17:35:23.387930 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.387880    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" podStartSLOduration=98.387865213 podStartE2EDuration="1m38.387865213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:23.386953565 +0000 UTC m=+99.045428565" watchObservedRunningTime="2026-09-17 17:35:23.387865213 +0000 UTC m=+99.046340213"
Sep 17 17:35:23.405831 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:23.405777    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bfb9fc68f-k6j6z" podStartSLOduration=2.280558471 podStartE2EDuration="5.405757946s" totalImagesPullingTime="3.125199475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:23.405106986 +0000 UTC m=+99.063581977" watchObservedRunningTime="2026-09-17 17:35:23.405757946 +0000 UTC m=+99.064232946"
Sep 17 17:35:25.378306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:25.378272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh8gs" event={"ID":"e0526a7e-2c2b-4892-ae86-883f3253ae5a","Type":"ContainerStarted","Data":"01498020d25be5b5c0e2e1688fbd8b8310cd09f5d27a985718a85f46e78c464f"}
Sep 17 17:35:25.378721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:25.378325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mh8gs" event={"ID":"e0526a7e-2c2b-4892-ae86-883f3253ae5a","Type":"ContainerStarted","Data":"8075dad82128c876779b265f400d28e83098ec628fc5475eb6601680db3e9db9"}
Sep 17 17:35:25.378721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:25.378417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:35:25.404582 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:25.404540    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mh8gs" podStartSLOduration=65.881877259 podStartE2EDuration="1m7.404527598s" totalImagesPullingTime="1.522650339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:25.400266507 +0000 UTC m=+101.058741509" watchObservedRunningTime="2026-09-17 17:35:25.404527598 +0000 UTC m=+101.063002597"
Sep 17 17:35:26.154601 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.154568    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2w9f5"]
Sep 17 17:35:26.157962 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.157940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.160113 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:35:26.160749 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:35:26.160749 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-twsbr\""
Sep 17 17:35:26.160749 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160622    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:35:26.160749 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160740    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:35:26.161032 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160767    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:35:26.161032 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.160871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:35:26.332547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-sys\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332656 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86rw\" (UniqueName: \"kubernetes.io/projected/171078fd-2465-47e0-80bd-6fa57d65baaa-kube-api-access-h86rw\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332656 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-metrics-client-ca\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-textfile\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332868 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-root\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332868 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-wtmp\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.332868 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.332853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.433675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-wtmp\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.433675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.433675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-sys\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86rw\" (UniqueName: \"kubernetes.io/projected/171078fd-2465-47e0-80bd-6fa57d65baaa-kube-api-access-h86rw\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-metrics-client-ca\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-textfile\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-wtmp\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-sys\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-root\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.433935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/171078fd-2465-47e0-80bd-6fa57d65baaa-root\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:35:26.434011    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:35:26.434096 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:35:26.434083    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls podName:171078fd-2465-47e0-80bd-6fa57d65baaa nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.934060158 +0000 UTC m=+102.592535135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls") pod "node-exporter-2w9f5" (UID: "171078fd-2465-47e0-80bd-6fa57d65baaa") : secret "node-exporter-tls" not found
Sep 17 17:35:26.434531 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.434136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-textfile\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434531 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.434287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.434531 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.434339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/171078fd-2465-47e0-80bd-6fa57d65baaa-metrics-client-ca\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.436220 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.436199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.446203 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.446173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86rw\" (UniqueName: \"kubernetes.io/projected/171078fd-2465-47e0-80bd-6fa57d65baaa-kube-api-access-h86rw\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.938331 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.938303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:26.940385 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:26.940359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/171078fd-2465-47e0-80bd-6fa57d65baaa-node-exporter-tls\") pod \"node-exporter-2w9f5\" (UID: \"171078fd-2465-47e0-80bd-6fa57d65baaa\") " pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:27.068729 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:27.068705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2w9f5"
Sep 17 17:35:27.080347 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:27.080322    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171078fd_2465_47e0_80bd_6fa57d65baaa.slice/crio-fbb8e57ebf625f168a064303586d01cde8cdcd1d1fcf7a1217a4580e1b05f73c WatchSource:0}: Error finding container fbb8e57ebf625f168a064303586d01cde8cdcd1d1fcf7a1217a4580e1b05f73c: Status 404 returned error can't find the container with id fbb8e57ebf625f168a064303586d01cde8cdcd1d1fcf7a1217a4580e1b05f73c
Sep 17 17:35:27.259661 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:27.259604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v424b"
Sep 17 17:35:27.391706 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:27.391673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2w9f5" event={"ID":"171078fd-2465-47e0-80bd-6fa57d65baaa","Type":"ContainerStarted","Data":"fbb8e57ebf625f168a064303586d01cde8cdcd1d1fcf7a1217a4580e1b05f73c"}
Sep 17 17:35:29.400378 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:29.400340    2577 generic.go:356] "Generic (PLEG): container finished" podID="171078fd-2465-47e0-80bd-6fa57d65baaa" containerID="1f70446d5034af65cf05b62b46b715982b5c3bad3f39e8751a2e4ddea6ca5a9f" exitCode=0
Sep 17 17:35:29.400738 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:29.400425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2w9f5" event={"ID":"171078fd-2465-47e0-80bd-6fa57d65baaa","Type":"ContainerDied","Data":"1f70446d5034af65cf05b62b46b715982b5c3bad3f39e8751a2e4ddea6ca5a9f"}
Sep 17 17:35:30.405953 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:30.405916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2w9f5" event={"ID":"171078fd-2465-47e0-80bd-6fa57d65baaa","Type":"ContainerStarted","Data":"87ef0e2388ff245e20372e2b4e8c2bdf6a600a58f1ec99bc4ab4f7f263020d35"}
Sep 17 17:35:30.406331 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:30.405972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2w9f5" event={"ID":"171078fd-2465-47e0-80bd-6fa57d65baaa","Type":"ContainerStarted","Data":"00f29ec2f6a568033f5859e5aa5b2e66e22aceb536758844175640f47186d4ba"}
Sep 17 17:35:30.426804 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:30.426754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2w9f5" podStartSLOduration=2.88595831 podStartE2EDuration="4.426739563s" totalImagesPullingTime="1.466298021s" totalInitContainerRuntime="74.483232ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:30.425072126 +0000 UTC m=+106.083547127" watchObservedRunningTime="2026-09-17 17:35:30.426739563 +0000 UTC m=+106.085214555"
Sep 17 17:35:35.384244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:35.384206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mh8gs"
Sep 17 17:35:35.845643 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:35.845560    2577 scope.go:117] "RemoveContainer" containerID="46bf1314d7ce38e89fa3de8efb066c4488fbe8f61dbb2ddb864f12286964c368"
Sep 17 17:35:36.427349 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:36.427324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:35:36.427706 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:36.427412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" event={"ID":"517ec90c-e584-4cd8-a7f0-f8a1e7207267","Type":"ContainerStarted","Data":"9c68162151cb99e34b04ad4494e131c9c0ab9b8c440d46f32a16f3bb7e9c9bad"}
Sep 17 17:35:36.427706 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:36.427690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:35:36.455437 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:36.455373    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d" podStartSLOduration=54.48761836 podStartE2EDuration="56.455358157s" totalImagesPullingTime="1.967739797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:36.454149071 +0000 UTC m=+112.112624063" watchObservedRunningTime="2026-09-17 17:35:36.455358157 +0000 UTC m=+112.113833157"
Sep 17 17:35:36.717224 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:36.717158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-cl69d"
Sep 17 17:35:41.899868 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:41.899835    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:35:41.904267 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:41.904239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:35:54.620918 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:54.620884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:35:54.623093 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:54.623072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d866f17-0e45-46eb-96c2-3638d706960c-metrics-certs\") pod \"network-metrics-daemon-gvfrz\" (UID: \"9d866f17-0e45-46eb-96c2-3638d706960c\") " pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:35:54.865290 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:54.865261    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2hx6\""
Sep 17 17:35:54.874069 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:54.874021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gvfrz"
Sep 17 17:35:55.204916 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:55.204843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gvfrz"]
Sep 17 17:35:55.209973 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:35:55.209945    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d866f17_0e45_46eb_96c2_3638d706960c.slice/crio-7b62276b98b7cd642eb4f6ef571b9a6a85cac644fafb2370c8f3102fe7379bd4 WatchSource:0}: Error finding container 7b62276b98b7cd642eb4f6ef571b9a6a85cac644fafb2370c8f3102fe7379bd4: Status 404 returned error can't find the container with id 7b62276b98b7cd642eb4f6ef571b9a6a85cac644fafb2370c8f3102fe7379bd4
Sep 17 17:35:55.492499 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:55.492418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvfrz" event={"ID":"9d866f17-0e45-46eb-96c2-3638d706960c","Type":"ContainerStarted","Data":"7b62276b98b7cd642eb4f6ef571b9a6a85cac644fafb2370c8f3102fe7379bd4"}
Sep 17 17:35:56.497744 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:56.497673    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvfrz" event={"ID":"9d866f17-0e45-46eb-96c2-3638d706960c","Type":"ContainerStarted","Data":"b1c1ba095cea930ae2a203a8e579d79dfc1a2be20393d24d3551835c4c282cce"}
Sep 17 17:35:56.497744 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:56.497731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gvfrz" event={"ID":"9d866f17-0e45-46eb-96c2-3638d706960c","Type":"ContainerStarted","Data":"83a9507937b9429eaee9dc3b160bea748de843047f9b06d3c50019f848ebf693"}
Sep 17 17:35:56.515639 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:35:56.515595    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gvfrz" podStartSLOduration=131.536322775 podStartE2EDuration="2m12.515578673s" totalImagesPullingTime="979.255898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:56.51475676 +0000 UTC m=+132.173231760" watchObservedRunningTime="2026-09-17 17:35:56.515578673 +0000 UTC m=+132.174053673"
Sep 17 17:36:06.918724 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:06.918664    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" podUID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" containerName="registry" containerID="cri-o://cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce" gracePeriod=30
Sep 17 17:36:07.157980 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.157949    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:36:07.199414 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199310    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199414 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199414 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199686 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199686 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199686 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199686 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.199877 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.199712    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vhlg\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg\") pod \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\" (UID: \"dbf374ae-adb7-4b3a-945c-abe14dd44caf\") "
Sep 17 17:36:07.200127 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.200096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.200315 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.200283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:36:07.202268 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.202198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:07.202373 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.202314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:07.202457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.202372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg" (OuterVolumeSpecName: "kube-api-access-7vhlg") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "kube-api-access-7vhlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:07.202520 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.202472    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:36:07.202520 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.202491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:36:07.208371 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.208343    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dbf374ae-adb7-4b3a-945c-abe14dd44caf" (UID: "dbf374ae-adb7-4b3a-945c-abe14dd44caf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:36:07.300906 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300874    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vhlg\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-kube-api-access-7vhlg\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.300906 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300900    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-tls\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.300906 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300910    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-installation-pull-secrets\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.301116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300918    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dbf374ae-adb7-4b3a-945c-abe14dd44caf-ca-trust-extracted\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.301116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300927    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dbf374ae-adb7-4b3a-945c-abe14dd44caf-image-registry-private-configuration\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.301116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300936    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-trusted-ca\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.301116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300944    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbf374ae-adb7-4b3a-945c-abe14dd44caf-bound-sa-token\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.301116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.300951    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dbf374ae-adb7-4b3a-945c-abe14dd44caf-registry-certificates\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:36:07.535227 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.535142    2577 generic.go:356] "Generic (PLEG): container finished" podID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" containerID="cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce" exitCode=0
Sep 17 17:36:07.535227 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.535188    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d984ffc-25jvh"
Sep 17 17:36:07.535227 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.535211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" event={"ID":"dbf374ae-adb7-4b3a-945c-abe14dd44caf","Type":"ContainerDied","Data":"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"}
Sep 17 17:36:07.535491 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.535251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d984ffc-25jvh" event={"ID":"dbf374ae-adb7-4b3a-945c-abe14dd44caf","Type":"ContainerDied","Data":"94ba768b465e1dae3e7d41295db4f2de5c3997404cfc3bb591920b22e38b5b22"}
Sep 17 17:36:07.535491 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.535270    2577 scope.go:117] "RemoveContainer" containerID="cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"
Sep 17 17:36:07.545457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.545434    2577 scope.go:117] "RemoveContainer" containerID="cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"
Sep 17 17:36:07.545872 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:36:07.545848    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce\": container with ID starting with cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce not found: ID does not exist" containerID="cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"
Sep 17 17:36:07.545953 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.545880    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce"} err="failed to get container status \"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce\": rpc error: code = NotFound desc = could not find container \"cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce\": container with ID starting with cf12c95c7d6e3a07b15c1b36f9805d8a123a50e46df56a8c20f6f624e03293ce not found: ID does not exist"
Sep 17 17:36:07.558469 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.558446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:36:07.564933 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:07.564909    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-846d984ffc-25jvh"]
Sep 17 17:36:08.848987 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:36:08.848947    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" path="/var/lib/kubelet/pods/dbf374ae-adb7-4b3a-945c-abe14dd44caf/volumes"
Sep 17 17:38:44.740865 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:38:44.740838    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:38:44.748925 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:38:44.748905    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:38:44.750173 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:38:44.750149    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:41:52.421620 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.421586    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"]
Sep 17 17:41:52.422024 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.421816    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" containerName="registry"
Sep 17 17:41:52.422024 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.421827    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" containerName="registry"
Sep 17 17:41:52.422024 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.421871    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf374ae-adb7-4b3a-945c-abe14dd44caf" containerName="registry"
Sep 17 17:41:52.424553 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.424536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.427206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.427173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-w74kj\""
Sep 17 17:41:52.427326 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.427230    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:41:52.427583 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.427566    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:41:52.427900 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.427882    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:41:52.440890 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.440869    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"]
Sep 17 17:41:52.483388 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.483370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17285c04-7802-4a16-9a1c-d53c70b44811-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.483491 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.483431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjvw\" (UniqueName: \"kubernetes.io/projected/17285c04-7802-4a16-9a1c-d53c70b44811-kube-api-access-tmjvw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.584569 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.584544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjvw\" (UniqueName: \"kubernetes.io/projected/17285c04-7802-4a16-9a1c-d53c70b44811-kube-api-access-tmjvw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.584675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.584597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17285c04-7802-4a16-9a1c-d53c70b44811-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.586812 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.586795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/17285c04-7802-4a16-9a1c-d53c70b44811-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.593624 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.593605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjvw\" (UniqueName: \"kubernetes.io/projected/17285c04-7802-4a16-9a1c-d53c70b44811-kube-api-access-tmjvw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-psgx2\" (UID: \"17285c04-7802-4a16-9a1c-d53c70b44811\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:52.735925 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:52.735840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:53.016032 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:53.015948    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"]
Sep 17 17:41:53.019306 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:41:53.019269    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17285c04_7802_4a16_9a1c_d53c70b44811.slice/crio-7dc77d3f03457f4ae12cfbeb0ac2c026da9bb24856acdb56df22c0e16ecf9d6b WatchSource:0}: Error finding container 7dc77d3f03457f4ae12cfbeb0ac2c026da9bb24856acdb56df22c0e16ecf9d6b: Status 404 returned error can't find the container with id 7dc77d3f03457f4ae12cfbeb0ac2c026da9bb24856acdb56df22c0e16ecf9d6b
Sep 17 17:41:53.021090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:53.021064    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:53.702966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:53.702936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2" event={"ID":"17285c04-7802-4a16-9a1c-d53c70b44811","Type":"ContainerStarted","Data":"7dc77d3f03457f4ae12cfbeb0ac2c026da9bb24856acdb56df22c0e16ecf9d6b"}
Sep 17 17:41:56.716684 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:56.716657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2" event={"ID":"17285c04-7802-4a16-9a1c-d53c70b44811","Type":"ContainerStarted","Data":"eaf3ce9938863011107e0d28be2e59c209db3ec35482942980578429a094922c"}
Sep 17 17:41:56.717001 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:56.716768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:41:56.735649 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:56.735606    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2" podStartSLOduration=1.118078855 podStartE2EDuration="4.735593719s" totalImagesPullingTime="3.617514864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:56.73422675 +0000 UTC m=+492.392702005" watchObservedRunningTime="2026-09-17 17:41:56.735593719 +0000 UTC m=+492.394068718"
Sep 17 17:41:57.332294 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.332257    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzhw"]
Sep 17 17:41:57.335357 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.335335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.337462 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.337439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:41:57.337907 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.337886    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 17:41:57.337994 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.337919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-wrfqw\""
Sep 17 17:41:57.346572 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.346551    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzhw"]
Sep 17 17:41:57.420302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.420271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-cabundle0\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.420450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.420316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c6h\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-kube-api-access-t6c6h\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.420450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.420411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.521360 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.521323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c6h\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-kube-api-access-t6c6h\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.521547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.521409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.521547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.521433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-cabundle0\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.521664 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:41:57.521546    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:41:57.521664 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:41:57.521565    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:41:57.521664 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:41:57.521577    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-jtzhw: references non-existent secret key: ca.crt
Sep 17 17:41:57.521664 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:41:57.521633    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates podName:a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:58.021617564 +0000 UTC m=+493.680092549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates") pod "keda-operator-579c8958f7-jtzhw" (UID: "a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2") : references non-existent secret key: ca.crt
Sep 17 17:41:57.522102 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.522080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-cabundle0\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:57.530888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:57.530864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c6h\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-kube-api-access-t6c6h\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:58.024765 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:58.024731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:58.027008 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:58.026985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2-certificates\") pod \"keda-operator-579c8958f7-jtzhw\" (UID: \"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2\") " pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:58.245461 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:58.245438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:41:58.516220 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:58.516190    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-jtzhw"]
Sep 17 17:41:58.519249 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:41:58.519222    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dbf9c2_6c81_4ac7_bd34_2ee8f49be1f2.slice/crio-13fb3c63bdd0d00c99c786e8c4bc1945d0f9ae65a0e15fdce1d75070444ad6e7 WatchSource:0}: Error finding container 13fb3c63bdd0d00c99c786e8c4bc1945d0f9ae65a0e15fdce1d75070444ad6e7: Status 404 returned error can't find the container with id 13fb3c63bdd0d00c99c786e8c4bc1945d0f9ae65a0e15fdce1d75070444ad6e7
Sep 17 17:41:58.726010 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:41:58.725978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-jtzhw" event={"ID":"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2","Type":"ContainerStarted","Data":"13fb3c63bdd0d00c99c786e8c4bc1945d0f9ae65a0e15fdce1d75070444ad6e7"}
Sep 17 17:42:01.739440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:42:01.739387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-jtzhw" event={"ID":"a6dbf9c2-6c81-4ac7-bd34-2ee8f49be1f2","Type":"ContainerStarted","Data":"3d83e032d446ee8ce3fcc9ac354427206b4e124fd558d6d6607c2095feaa7c56"}
Sep 17 17:42:01.739829 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:42:01.739530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:42:01.756763 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:42:01.756718    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-jtzhw" podStartSLOduration=1.951881281 podStartE2EDuration="4.756702398s" totalImagesPullingTime="2.804821117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:01.7558899 +0000 UTC m=+497.414364902" watchObservedRunningTime="2026-09-17 17:42:01.756702398 +0000 UTC m=+497.415177432"
Sep 17 17:42:17.721653 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:42:17.721624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-psgx2"
Sep 17 17:42:22.745274 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:42:22.745243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-jtzhw"
Sep 17 17:43:03.099154 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.099069    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"]
Sep 17 17:43:03.101049 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.101032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.104457 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.104438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m4w6k\""
Sep 17 17:43:03.104587 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.104558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:43:03.104791 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.104777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 17:43:03.118756 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.118732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"]
Sep 17 17:43:03.163387 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.163350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdprt\" (UniqueName: \"kubernetes.io/projected/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-kube-api-access-tdprt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.163578 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.163423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.263970 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.263930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.264106 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.263990    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdprt\" (UniqueName: \"kubernetes.io/projected/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-kube-api-access-tdprt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.264305 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.264286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.273222 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.273203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdprt\" (UniqueName: \"kubernetes.io/projected/1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7-kube-api-access-tdprt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-sh7c2\" (UID: \"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.410020 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.409989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"
Sep 17 17:43:03.682691 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.682601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2"]
Sep 17 17:43:03.685689 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:43:03.685664    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1174fb34_b1b5_4ae1_aee5_14bc7c7fbee7.slice/crio-a61cdb2d3dd4420d02f4c9d3dfb2d62498354b524af170433160aa4716630809 WatchSource:0}: Error finding container a61cdb2d3dd4420d02f4c9d3dfb2d62498354b524af170433160aa4716630809: Status 404 returned error can't find the container with id a61cdb2d3dd4420d02f4c9d3dfb2d62498354b524af170433160aa4716630809
Sep 17 17:43:03.945988 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:03.945907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2" event={"ID":"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7","Type":"ContainerStarted","Data":"a61cdb2d3dd4420d02f4c9d3dfb2d62498354b524af170433160aa4716630809"}
Sep 17 17:43:06.961445 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:06.961383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2" event={"ID":"1174fb34-b1b5-4ae1-aee5-14bc7c7fbee7","Type":"ContainerStarted","Data":"07bce9f107e4856646a75ddfe7176121696089e61436d5195098d1838a90d3db"}
Sep 17 17:43:06.982306 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:06.982253    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-sh7c2" podStartSLOduration=1.170910608 podStartE2EDuration="3.982240855s" totalImagesPullingTime="2.811330247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:06.981192436 +0000 UTC m=+562.639667436" watchObservedRunningTime="2026-09-17 17:43:06.982240855 +0000 UTC m=+562.640715855"
Sep 17 17:43:24.840793 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.840758    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"]
Sep 17 17:43:24.843616 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.843598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:24.846125 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.846092    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:43:24.846252 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.846126    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2bkjt\""
Sep 17 17:43:24.846770 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.846756    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:43:24.852002 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.851978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"]
Sep 17 17:43:24.913785 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.913763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a6398b-dce4-4c15-8848-dfc10187952b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:24.913915 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:24.913803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8gp\" (UniqueName: \"kubernetes.io/projected/77a6398b-dce4-4c15-8848-dfc10187952b-kube-api-access-7p8gp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.014576 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.014549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a6398b-dce4-4c15-8848-dfc10187952b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.014695 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.014582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8gp\" (UniqueName: \"kubernetes.io/projected/77a6398b-dce4-4c15-8848-dfc10187952b-kube-api-access-7p8gp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.014914 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.014894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77a6398b-dce4-4c15-8848-dfc10187952b-tmp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.025341 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.025311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8gp\" (UniqueName: \"kubernetes.io/projected/77a6398b-dce4-4c15-8848-dfc10187952b-kube-api-access-7p8gp\") pod \"openshift-lws-operator-fd8ccff4c-2cwgm\" (UID: \"77a6398b-dce4-4c15-8848-dfc10187952b\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.153434 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.153412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"
Sep 17 17:43:25.427853 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:25.427784    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm"]
Sep 17 17:43:25.431075 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:43:25.431047    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77a6398b_dce4_4c15_8848_dfc10187952b.slice/crio-2f4ad6f8ea0c7168f08f76ee08ce1adcaeeac3ce53256cbc567eec77073970b6 WatchSource:0}: Error finding container 2f4ad6f8ea0c7168f08f76ee08ce1adcaeeac3ce53256cbc567eec77073970b6: Status 404 returned error can't find the container with id 2f4ad6f8ea0c7168f08f76ee08ce1adcaeeac3ce53256cbc567eec77073970b6
Sep 17 17:43:26.031032 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:26.030996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm" event={"ID":"77a6398b-dce4-4c15-8848-dfc10187952b","Type":"ContainerStarted","Data":"2f4ad6f8ea0c7168f08f76ee08ce1adcaeeac3ce53256cbc567eec77073970b6"}
Sep 17 17:43:29.046173 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:29.046137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm" event={"ID":"77a6398b-dce4-4c15-8848-dfc10187952b","Type":"ContainerStarted","Data":"7297b6ee5bf742d1fd71abe75db866d7e14f83e24362a4be7be7f61b5ebc174c"}
Sep 17 17:43:29.061992 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:29.061917    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-2cwgm" podStartSLOduration=2.19484024 podStartE2EDuration="5.061902906s" totalImagesPullingTime="2.867062666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:29.061306077 +0000 UTC m=+584.719781078" watchObservedRunningTime="2026-09-17 17:43:29.061902906 +0000 UTC m=+584.720377906"
Sep 17 17:43:40.034073 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.034038    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:43:40.068047 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.068014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:43:40.068189 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.068108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.070590 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.070560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 17:43:40.070716 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.070604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 17:43:40.070950 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.070935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2xclc\""
Sep 17 17:43:40.071167 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.071145    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:43:40.071167 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.071163    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 17:43:40.071294 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.071194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 17:43:40.071294 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.071244    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 17:43:40.122979 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.122936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123126 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.122990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123126 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.123020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123126 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.123083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123126 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.123114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.123129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9fjs\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.123302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.123187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9fjs\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224547 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.224760 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.224548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.225141 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.225061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.226796 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.226773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.226880 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.226800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.226984 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.226966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.227118 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.227099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.233651 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.233623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.234156 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.234125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9fjs\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs\") pod \"istiod-openshift-gateway-7b7995d5cd-47nw9\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.377375 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.377342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:40.652293 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:40.652216    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:43:40.656776 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:43:40.656749    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190b85fe_f995_45b2_aac0_acac20ffd771.slice/crio-e376e4eb55cc4d877109096c552dc9340ac71094516df39443a8f7a37ba60aa6 WatchSource:0}: Error finding container e376e4eb55cc4d877109096c552dc9340ac71094516df39443a8f7a37ba60aa6: Status 404 returned error can't find the container with id e376e4eb55cc4d877109096c552dc9340ac71094516df39443a8f7a37ba60aa6
Sep 17 17:43:41.095532 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:41.095449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" event={"ID":"190b85fe-f995-45b2-aac0-acac20ffd771","Type":"ContainerStarted","Data":"e376e4eb55cc4d877109096c552dc9340ac71094516df39443a8f7a37ba60aa6"}
Sep 17 17:43:43.371362 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:43.371320    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:43:43.371653 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:43.371420    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:43:44.110780 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.110744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" event={"ID":"190b85fe-f995-45b2-aac0-acac20ffd771","Type":"ContainerStarted","Data":"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"}
Sep 17 17:43:44.110967 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.110914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:43:44.112769 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.112738    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-47nw9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 17:43:44.112878 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.112791    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:43:44.130710 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.130665    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" podStartSLOduration=1.418735023 podStartE2EDuration="4.130651264s" totalImagesPullingTime="2.711916241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:44.13046908 +0000 UTC m=+599.788944075" watchObservedRunningTime="2026-09-17 17:43:44.130651264 +0000 UTC m=+599.789126263"
Sep 17 17:43:44.766874 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.766847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:43:44.770604 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:44.770581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:43:45.114891 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:43:45.114856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:44:09.750356 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.750271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"]
Sep 17 17:44:09.753528 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.753506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:09.756183 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.756164    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:44:09.756814 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.756798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:44:09.756873 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.756828    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b26c7\""
Sep 17 17:44:09.765564 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.765544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"]
Sep 17 17:44:09.839483 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.839450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvf57\" (UniqueName: \"kubernetes.io/projected/09aebdaf-0437-4c63-9995-d7f7da1b5643-kube-api-access-dvf57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wvl5v\" (UID: \"09aebdaf-0437-4c63-9995-d7f7da1b5643\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:09.940135 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.940097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvf57\" (UniqueName: \"kubernetes.io/projected/09aebdaf-0437-4c63-9995-d7f7da1b5643-kube-api-access-dvf57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wvl5v\" (UID: \"09aebdaf-0437-4c63-9995-d7f7da1b5643\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:09.953773 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:09.953744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvf57\" (UniqueName: \"kubernetes.io/projected/09aebdaf-0437-4c63-9995-d7f7da1b5643-kube-api-access-dvf57\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-wvl5v\" (UID: \"09aebdaf-0437-4c63-9995-d7f7da1b5643\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:10.062953 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:10.062880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:10.356088 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:10.356054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"]
Sep 17 17:44:10.361025 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:44:10.360995    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aebdaf_0437_4c63_9995_d7f7da1b5643.slice/crio-20d06392a4096d85e039547e6d64e7c3b26d64839bed916a5fa86d54c523de4d WatchSource:0}: Error finding container 20d06392a4096d85e039547e6d64e7c3b26d64839bed916a5fa86d54c523de4d: Status 404 returned error can't find the container with id 20d06392a4096d85e039547e6d64e7c3b26d64839bed916a5fa86d54c523de4d
Sep 17 17:44:11.218876 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.218829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v" event={"ID":"09aebdaf-0437-4c63-9995-d7f7da1b5643","Type":"ContainerStarted","Data":"20d06392a4096d85e039547e6d64e7c3b26d64839bed916a5fa86d54c523de4d"}
Sep 17 17:44:11.924232 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.924189    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"]
Sep 17 17:44:11.931004 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.930977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:11.933144 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.933120    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 17:44:11.933407 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.933378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dd75f\""
Sep 17 17:44:11.952074 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:11.952046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"]
Sep 17 17:44:12.057675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:12.057642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctccj\" (UniqueName: \"kubernetes.io/projected/9cea7ecb-c655-4d21-b737-8c8a8e1985cb-kube-api-access-ctccj\") pod \"dns-operator-controller-manager-65dc8f76f5-mdg6t\" (UID: \"9cea7ecb-c655-4d21-b737-8c8a8e1985cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:12.158560 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:12.158525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctccj\" (UniqueName: \"kubernetes.io/projected/9cea7ecb-c655-4d21-b737-8c8a8e1985cb-kube-api-access-ctccj\") pod \"dns-operator-controller-manager-65dc8f76f5-mdg6t\" (UID: \"9cea7ecb-c655-4d21-b737-8c8a8e1985cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:12.176025 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:12.175958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctccj\" (UniqueName: \"kubernetes.io/projected/9cea7ecb-c655-4d21-b737-8c8a8e1985cb-kube-api-access-ctccj\") pod \"dns-operator-controller-manager-65dc8f76f5-mdg6t\" (UID: \"9cea7ecb-c655-4d21-b737-8c8a8e1985cb\") " pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:12.242795 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:12.242776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:12.523525 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:12.523330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"]
Sep 17 17:44:12.526118 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:44:12.526087    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cea7ecb_c655_4d21_b737_8c8a8e1985cb.slice/crio-b87318bd8e5c3ade4e1ddad6eb00689ee56ef9782604b11f99c5e05897598845 WatchSource:0}: Error finding container b87318bd8e5c3ade4e1ddad6eb00689ee56ef9782604b11f99c5e05897598845: Status 404 returned error can't find the container with id b87318bd8e5c3ade4e1ddad6eb00689ee56ef9782604b11f99c5e05897598845
Sep 17 17:44:13.228601 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:13.228556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t" event={"ID":"9cea7ecb-c655-4d21-b737-8c8a8e1985cb","Type":"ContainerStarted","Data":"b87318bd8e5c3ade4e1ddad6eb00689ee56ef9782604b11f99c5e05897598845"}
Sep 17 17:44:13.230321 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:13.230289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v" event={"ID":"09aebdaf-0437-4c63-9995-d7f7da1b5643","Type":"ContainerStarted","Data":"127da7129a06a6a63b45eb9b9d16fac958a8b9adf76917deae51f11080fc9c11"}
Sep 17 17:44:13.230469 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:13.230433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:13.261837 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:13.261788    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v" podStartSLOduration=2.420316675 podStartE2EDuration="4.261771002s" totalImagesPullingTime="1.841454327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:13.260465058 +0000 UTC m=+628.918940058" watchObservedRunningTime="2026-09-17 17:44:13.261771002 +0000 UTC m=+628.920246002"
Sep 17 17:44:14.237031 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.237003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t" event={"ID":"9cea7ecb-c655-4d21-b737-8c8a8e1985cb","Type":"ContainerStarted","Data":"90af6c08cf06abb32f200bb92ef8210538917a7efd76af5d1c596473766e2645"}
Sep 17 17:44:14.737296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.737255    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"]
Sep 17 17:44:14.740683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.740667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:14.743456 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.743434    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9drgb\""
Sep 17 17:44:14.815484 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.815461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"]
Sep 17 17:44:14.878229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.878203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ptp\" (UniqueName: \"kubernetes.io/projected/32401980-5604-4d44-8188-1c5e904ce975-kube-api-access-g2ptp\") pod \"authorino-operator-6fc8f864f4-qqzjm\" (UID: \"32401980-5604-4d44-8188-1c5e904ce975\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:14.979073 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.979049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ptp\" (UniqueName: \"kubernetes.io/projected/32401980-5604-4d44-8188-1c5e904ce975-kube-api-access-g2ptp\") pod \"authorino-operator-6fc8f864f4-qqzjm\" (UID: \"32401980-5604-4d44-8188-1c5e904ce975\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:14.992568 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:14.992514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ptp\" (UniqueName: \"kubernetes.io/projected/32401980-5604-4d44-8188-1c5e904ce975-kube-api-access-g2ptp\") pod \"authorino-operator-6fc8f864f4-qqzjm\" (UID: \"32401980-5604-4d44-8188-1c5e904ce975\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:15.051251 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:15.051230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:15.241658 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:15.241623    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:15.260943 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:15.260864    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t" podStartSLOduration=2.611210058 podStartE2EDuration="4.260846749s" totalImagesPullingTime="1.649636691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:15.259546425 +0000 UTC m=+630.918021425" watchObservedRunningTime="2026-09-17 17:44:15.260846749 +0000 UTC m=+630.919321750"
Sep 17 17:44:15.326202 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:15.326171    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"]
Sep 17 17:44:15.329674 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:44:15.329649    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32401980_5604_4d44_8188_1c5e904ce975.slice/crio-d83134cb6c0ac915aba48f6b54368775d05119c62af2320b6b1d15281b7aaa6c WatchSource:0}: Error finding container d83134cb6c0ac915aba48f6b54368775d05119c62af2320b6b1d15281b7aaa6c: Status 404 returned error can't find the container with id d83134cb6c0ac915aba48f6b54368775d05119c62af2320b6b1d15281b7aaa6c
Sep 17 17:44:16.247981 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:16.247939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm" event={"ID":"32401980-5604-4d44-8188-1c5e904ce975","Type":"ContainerStarted","Data":"d83134cb6c0ac915aba48f6b54368775d05119c62af2320b6b1d15281b7aaa6c"}
Sep 17 17:44:17.253963 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:17.253926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm" event={"ID":"32401980-5604-4d44-8188-1c5e904ce975","Type":"ContainerStarted","Data":"4aff8c8fa2571df334c0d5ae289e46e30c82574ebccc1a20873ed2e96378c97d"}
Sep 17 17:44:17.254333 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:17.254035    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:17.282463 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:17.282418    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm" podStartSLOduration=1.786845756 podStartE2EDuration="3.282387102s" totalImagesPullingTime="1.495541346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:17.280922708 +0000 UTC m=+632.939397709" watchObservedRunningTime="2026-09-17 17:44:17.282387102 +0000 UTC m=+632.940862102"
Sep 17 17:44:19.594210 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.594171    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"]
Sep 17 17:44:19.602440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.602413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.612902 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.612878    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pkcg6\""
Sep 17 17:44:19.614310 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.614285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"]
Sep 17 17:44:19.716496 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.716468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxf4\" (UniqueName: \"kubernetes.io/projected/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-kube-api-access-lbxf4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.716592 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.716508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.817781 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.817746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxf4\" (UniqueName: \"kubernetes.io/projected/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-kube-api-access-lbxf4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.817781 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.817787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.818083 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.818069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.829731 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.829705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxf4\" (UniqueName: \"kubernetes.io/projected/7ddc4719-0aa1-4d32-980a-eeacf456ec0b-kube-api-access-lbxf4\") pod \"kuadrant-operator-controller-manager-6d4cb79c-cqkdm\" (UID: \"7ddc4719-0aa1-4d32-980a-eeacf456ec0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:19.912261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:19.912238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:20.193539 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:20.193516    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"]
Sep 17 17:44:20.195677 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:44:20.195647    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddc4719_0aa1_4d32_980a_eeacf456ec0b.slice/crio-6b6fe674311426d0c7932788554d39f1bee02c4a6a749232c9442749112b9df7 WatchSource:0}: Error finding container 6b6fe674311426d0c7932788554d39f1bee02c4a6a749232c9442749112b9df7: Status 404 returned error can't find the container with id 6b6fe674311426d0c7932788554d39f1bee02c4a6a749232c9442749112b9df7
Sep 17 17:44:20.268839 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:20.268809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm" event={"ID":"7ddc4719-0aa1-4d32-980a-eeacf456ec0b","Type":"ContainerStarted","Data":"6b6fe674311426d0c7932788554d39f1bee02c4a6a749232c9442749112b9df7"}
Sep 17 17:44:24.239358 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:24.239321    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-wvl5v"
Sep 17 17:44:25.294463 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:25.294428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm" event={"ID":"7ddc4719-0aa1-4d32-980a-eeacf456ec0b","Type":"ContainerStarted","Data":"fbe9f11c8db12346e54f51078ebf907552c2c876caba0a771f9deec279faf66c"}
Sep 17 17:44:25.294847 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:25.294506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:44:25.319426 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:25.319366    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm" podStartSLOduration=2.105707828 podStartE2EDuration="6.31935014s" totalImagesPullingTime="4.213642312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:25.317055832 +0000 UTC m=+640.975530832" watchObservedRunningTime="2026-09-17 17:44:25.31935014 +0000 UTC m=+640.977825141"
Sep 17 17:44:26.250358 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:26.250327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65dc8f76f5-mdg6t"
Sep 17 17:44:28.260386 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:28.260355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-qqzjm"
Sep 17 17:44:36.301623 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:44:36.301520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-cqkdm"
Sep 17 17:45:55.268809 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.268733    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"]
Sep 17 17:45:55.270800 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.270783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.282158 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.282135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"]
Sep 17 17:45:55.421798 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.421798 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.421972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6ec2e3-4301-4131-b718-69fb39c67bb7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.421972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.421972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.421972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgh5d\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-kube-api-access-cgh5d\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.422126 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.421992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523042 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.522969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523042 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523042 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6ec2e3-4301-4131-b718-69fb39c67bb7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgh5d\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-kube-api-access-cgh5d\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523279 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.523945 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.523918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.525528 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.525492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6ec2e3-4301-4131-b718-69fb39c67bb7-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.525780 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.525756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.526062 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.526042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.526141 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.526123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.532169 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.532139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.532822 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.532798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgh5d\" (UniqueName: \"kubernetes.io/projected/be6ec2e3-4301-4131-b718-69fb39c67bb7-kube-api-access-cgh5d\") pod \"istiod-openshift-gateway-5677ccb49b-d82ns\" (UID: \"be6ec2e3-4301-4131-b718-69fb39c67bb7\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.579648 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.579625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:55.859379 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.859351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"]
Sep 17 17:45:55.862613 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:45:55.862578    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6ec2e3_4301_4131_b718_69fb39c67bb7.slice/crio-2b99e14e21bd2b40cc805ae0524c392b471c931c1eb7aad3bb6e93ca01a7543d WatchSource:0}: Error finding container 2b99e14e21bd2b40cc805ae0524c392b471c931c1eb7aad3bb6e93ca01a7543d: Status 404 returned error can't find the container with id 2b99e14e21bd2b40cc805ae0524c392b471c931c1eb7aad3bb6e93ca01a7543d
Sep 17 17:45:55.864692 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.864657    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:45:55.864778 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:55.864730    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:45:56.690568 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.690532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns" event={"ID":"be6ec2e3-4301-4131-b718-69fb39c67bb7","Type":"ContainerStarted","Data":"690ed9be8bb9cfcfefb00829d1d98a78fc03191e9f128c8d94c1fde173d0a8f6"}
Sep 17 17:45:56.690972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.690589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns" event={"ID":"be6ec2e3-4301-4131-b718-69fb39c67bb7","Type":"ContainerStarted","Data":"2b99e14e21bd2b40cc805ae0524c392b471c931c1eb7aad3bb6e93ca01a7543d"}
Sep 17 17:45:56.690972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.690770    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:56.692150 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.692116    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-d82ns container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 17:45:56.692284 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.692168    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns" podUID="be6ec2e3-4301-4131-b718-69fb39c67bb7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:45:56.713660 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:56.713616    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns" podStartSLOduration=1.713602121 podStartE2EDuration="1.713602121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:56.712416128 +0000 UTC m=+732.370891129" watchObservedRunningTime="2026-09-17 17:45:56.713602121 +0000 UTC m=+732.372077120"
Sep 17 17:45:57.696064 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:57.696031    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-d82ns"
Sep 17 17:45:57.752627 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:57.752599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:45:57.752797 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:57.752776    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" containerName="discovery" containerID="cri-o://37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099" gracePeriod=30
Sep 17 17:45:58.002323 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.002300    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:45:58.145385 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145548 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9fjs\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145548 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145548 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145548 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145520    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145760 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.145760 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.145581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap\") pod \"190b85fe-f995-45b2-aac0-acac20ffd771\" (UID: \"190b85fe-f995-45b2-aac0-acac20ffd771\") "
Sep 17 17:45:58.146112 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.146079    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:45:58.147835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.147738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs" (OuterVolumeSpecName: "local-certs") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:45:58.147835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.147798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token" (OuterVolumeSpecName: "istio-token") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:58.147980 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.147897    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:58.148060 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.148044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:58.148098 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.148069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts" (OuterVolumeSpecName: "cacerts") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:45:58.148132 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.148099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs" (OuterVolumeSpecName: "kube-api-access-z9fjs") pod "190b85fe-f995-45b2-aac0-acac20ffd771" (UID: "190b85fe-f995-45b2-aac0-acac20ffd771"). InnerVolumeSpecName "kube-api-access-z9fjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:45:58.246440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246352    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/190b85fe-f995-45b2-aac0-acac20ffd771-local-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246381    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-kubeconfig\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246422    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-cacerts\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246440 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246436    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-dns-cert\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246629 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246448    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/190b85fe-f995-45b2-aac0-acac20ffd771-istio-csr-ca-configmap\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246629 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246458    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-istio-token\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.246629 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.246466    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9fjs\" (UniqueName: \"kubernetes.io/projected/190b85fe-f995-45b2-aac0-acac20ffd771-kube-api-access-z9fjs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:45:58.701112 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.701077    2577 generic.go:356] "Generic (PLEG): container finished" podID="190b85fe-f995-45b2-aac0-acac20ffd771" containerID="37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099" exitCode=0
Sep 17 17:45:58.701533 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.701129    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"
Sep 17 17:45:58.701533 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.701166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" event={"ID":"190b85fe-f995-45b2-aac0-acac20ffd771","Type":"ContainerDied","Data":"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"}
Sep 17 17:45:58.701533 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.701216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9" event={"ID":"190b85fe-f995-45b2-aac0-acac20ffd771","Type":"ContainerDied","Data":"e376e4eb55cc4d877109096c552dc9340ac71094516df39443a8f7a37ba60aa6"}
Sep 17 17:45:58.701533 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.701236    2577 scope.go:117] "RemoveContainer" containerID="37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"
Sep 17 17:45:58.709650 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.709633    2577 scope.go:117] "RemoveContainer" containerID="37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"
Sep 17 17:45:58.709918 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:45:58.709902    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099\": container with ID starting with 37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099 not found: ID does not exist" containerID="37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"
Sep 17 17:45:58.709956 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.709926    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099"} err="failed to get container status \"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099\": rpc error: code = NotFound desc = could not find container \"37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099\": container with ID starting with 37d23e3c63c25fdf894b0ff28b25fb2c58fedcf9e0de839f25e7e458a6971099 not found: ID does not exist"
Sep 17 17:45:58.722423 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.722387    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:45:58.726471 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.726449    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-47nw9"]
Sep 17 17:45:58.851275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:45:58.851239    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" path="/var/lib/kubelet/pods/190b85fe-f995-45b2-aac0-acac20ffd771/volumes"
Sep 17 17:47:24.820474 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.820439    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-gzlsj"]
Sep 17 17:47:24.820916 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.820740    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" containerName="discovery"
Sep 17 17:47:24.820916 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.820752    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" containerName="discovery"
Sep 17 17:47:24.820916 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.820807    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="190b85fe-f995-45b2-aac0-acac20ffd771" containerName="discovery"
Sep 17 17:47:24.823570 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.823554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:24.825849 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.825825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-nl64g\""
Sep 17 17:47:24.825987 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.825830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:47:24.826318 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.826302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:47:24.826441 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.826361    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:47:24.830916 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.830892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-gzlsj"]
Sep 17 17:47:24.835231 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.835211    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-pjt6r"]
Sep 17 17:47:24.838776 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.838756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:24.841004 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.840983    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mzjbv\""
Sep 17 17:47:24.841108 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.841027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 17:47:24.841170 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.840984    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 17:47:24.854519 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.854271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-pjt6r"]
Sep 17 17:47:24.961872 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.961839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5v9\" (UniqueName: \"kubernetes.io/projected/cfae0daa-f955-4c8d-a10b-b12e93762c25-kube-api-access-rc5v9\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:24.961872 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.961872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:24.962051 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.961904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfae0daa-f955-4c8d-a10b-b12e93762c25-tls-certs\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:24.962051 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.962000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-metrics-certs\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:24.962051 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:24.962033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mkd\" (UniqueName: \"kubernetes.io/projected/28c081b6-9bd3-48ae-bb97-27f08e316b50-kube-api-access-67mkd\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.063057 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.063015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfae0daa-f955-4c8d-a10b-b12e93762c25-tls-certs\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:25.063229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.063068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-metrics-certs\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.063229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.063087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mkd\" (UniqueName: \"kubernetes.io/projected/28c081b6-9bd3-48ae-bb97-27f08e316b50-kube-api-access-67mkd\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.063229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.063135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5v9\" (UniqueName: \"kubernetes.io/projected/cfae0daa-f955-4c8d-a10b-b12e93762c25-kube-api-access-rc5v9\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:25.063229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.063153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.063426 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:47:25.063256    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 17:47:25.063426 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:47:25.063344    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert podName:28c081b6-9bd3-48ae-bb97-27f08e316b50 nodeName:}" failed. No retries permitted until 2026-09-17 17:47:25.563327556 +0000 UTC m=+821.221802534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert") pod "odh-model-controller-7b966d7984-pjt6r" (UID: "28c081b6-9bd3-48ae-bb97-27f08e316b50") : secret "odh-model-controller-webhook-cert" not found
Sep 17 17:47:25.065896 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.065871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfae0daa-f955-4c8d-a10b-b12e93762c25-tls-certs\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:25.066006 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.065895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-metrics-certs\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.075288 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.075231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mkd\" (UniqueName: \"kubernetes.io/projected/28c081b6-9bd3-48ae-bb97-27f08e316b50-kube-api-access-67mkd\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.075438 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.075420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5v9\" (UniqueName: \"kubernetes.io/projected/cfae0daa-f955-4c8d-a10b-b12e93762c25-kube-api-access-rc5v9\") pod \"model-serving-api-7b954ffc95-gzlsj\" (UID: \"cfae0daa-f955-4c8d-a10b-b12e93762c25\") " pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:25.137230 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.137192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:25.408688 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.408648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-gzlsj"]
Sep 17 17:47:25.411465 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:47:25.411440    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfae0daa_f955_4c8d_a10b_b12e93762c25.slice/crio-a4806d00d8fc2ba6d2a2d53b4b247ad4f25c9c9d35c8c6f1f3da597070309115 WatchSource:0}: Error finding container a4806d00d8fc2ba6d2a2d53b4b247ad4f25c9c9d35c8c6f1f3da597070309115: Status 404 returned error can't find the container with id a4806d00d8fc2ba6d2a2d53b4b247ad4f25c9c9d35c8c6f1f3da597070309115
Sep 17 17:47:25.413257 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.413241    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:25.568149 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.568122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.570429 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.570407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28c081b6-9bd3-48ae-bb97-27f08e316b50-cert\") pod \"odh-model-controller-7b966d7984-pjt6r\" (UID: \"28c081b6-9bd3-48ae-bb97-27f08e316b50\") " pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:25.752372 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:25.752284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:26.024981 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:26.024818    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-pjt6r"]
Sep 17 17:47:26.027322 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:47:26.027298    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c081b6_9bd3_48ae_bb97_27f08e316b50.slice/crio-f923e3f718f776194f5210a2e88347af25b163cfb52ae4e229c8d600b195e7d2 WatchSource:0}: Error finding container f923e3f718f776194f5210a2e88347af25b163cfb52ae4e229c8d600b195e7d2: Status 404 returned error can't find the container with id f923e3f718f776194f5210a2e88347af25b163cfb52ae4e229c8d600b195e7d2
Sep 17 17:47:26.103679 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:26.103640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-gzlsj" event={"ID":"cfae0daa-f955-4c8d-a10b-b12e93762c25","Type":"ContainerStarted","Data":"a4806d00d8fc2ba6d2a2d53b4b247ad4f25c9c9d35c8c6f1f3da597070309115"}
Sep 17 17:47:26.105016 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:26.104993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-pjt6r" event={"ID":"28c081b6-9bd3-48ae-bb97-27f08e316b50","Type":"ContainerStarted","Data":"f923e3f718f776194f5210a2e88347af25b163cfb52ae4e229c8d600b195e7d2"}
Sep 17 17:47:28.118574 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:28.118526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-gzlsj" event={"ID":"cfae0daa-f955-4c8d-a10b-b12e93762c25","Type":"ContainerStarted","Data":"3e0714d34a055c2e6b71a0cde13a37175beb25aeb4cbb53b15725e654c766636"}
Sep 17 17:47:28.119059 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:28.118635    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:28.135621 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:28.135567    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-gzlsj" podStartSLOduration=1.787843947 podStartE2EDuration="4.135546604s" totalImagesPullingTime="2.347702657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:28.133903517 +0000 UTC m=+823.792378531" watchObservedRunningTime="2026-09-17 17:47:28.135546604 +0000 UTC m=+823.794021605"
Sep 17 17:47:31.134983 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:31.134950    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-pjt6r" event={"ID":"28c081b6-9bd3-48ae-bb97-27f08e316b50","Type":"ContainerStarted","Data":"090f9b61d54f8e292734676e43176c7aee96f002b130224a0724599e68d01c70"}
Sep 17 17:47:31.135433 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:31.135075    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:47:31.154729 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:31.154675    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-pjt6r" podStartSLOduration=3.10480218 podStartE2EDuration="7.154657254s" totalImagesPullingTime="4.049855074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:31.153105141 +0000 UTC m=+826.811580140" watchObservedRunningTime="2026-09-17 17:47:31.154657254 +0000 UTC m=+826.813132255"
Sep 17 17:47:39.129218 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:39.129183    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-gzlsj"
Sep 17 17:47:42.141556 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:47:42.141519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-pjt6r"
Sep 17 17:48:44.792326 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:48:44.792245    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:48:44.794291 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:48:44.794266    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:49:24.758477 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.758445    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:49:24.761847 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.761830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.764975 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.764955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 17:49:24.765097 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.765050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-f85b5\""
Sep 17 17:49:24.765198 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.765160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jjghq\""
Sep 17 17:49:24.765334 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.765278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 17:49:24.765448 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.765427    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 17:49:24.771191 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.771166    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:49:24.866817 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.866793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6z6w\" (UniqueName: \"kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.866918 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.866837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.967367 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.967341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6z6w\" (UniqueName: \"kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.967552 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.967530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.969997 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.969974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:24.975945 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:24.975924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6z6w\" (UniqueName: \"kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:25.072888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:25.072835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:25.350274 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:25.350118    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:49:25.352913 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:49:25.352882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d53c4d2_f586_4c06_af71_c168e7280c59.slice/crio-4bd50d0bf1dd529b41dc319f5102b890371b20174541b58fc34d959a076141b7 WatchSource:0}: Error finding container 4bd50d0bf1dd529b41dc319f5102b890371b20174541b58fc34d959a076141b7: Status 404 returned error can't find the container with id 4bd50d0bf1dd529b41dc319f5102b890371b20174541b58fc34d959a076141b7
Sep 17 17:49:25.647549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:25.647516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" event={"ID":"4d53c4d2-f586-4c06-af71-c168e7280c59","Type":"ContainerStarted","Data":"4bd50d0bf1dd529b41dc319f5102b890371b20174541b58fc34d959a076141b7"}
Sep 17 17:49:28.666206 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:28.666171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" event={"ID":"4d53c4d2-f586-4c06-af71-c168e7280c59","Type":"ContainerStarted","Data":"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"}
Sep 17 17:49:28.666606 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:28.666243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:49:28.682259 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:28.682202    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podStartSLOduration=2.304426888 podStartE2EDuration="4.682183898s" totalImagesPullingTime="2.37775701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:28.682127386 +0000 UTC m=+944.340602392" watchObservedRunningTime="2026-09-17 17:49:28.682183898 +0000 UTC m=+944.340658897"
Sep 17 17:49:37.600504 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.600457    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:49:37.604163 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.604141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.606668 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.606646    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:49:37.607071 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.607048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:49:37.607179 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.607084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:49:37.607370 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.607351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-dwrjn\""
Sep 17 17:49:37.614342 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.614317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:49:37.654737 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.654696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.654737 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.654742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.654973 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.654768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvhk\" (UniqueName: \"kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.654973 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.654884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.654973 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.654931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.655104 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.655014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755530 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755530 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755768 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755768 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755768 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755768 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvhk\" (UniqueName: \"kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.755990 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.756043 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.755985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.756086 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.756070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.757988 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.757957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.758259 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.758239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.766598 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.766568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvhk\" (UniqueName: \"kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-m84hf\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:37.915912 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:37.915882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:38.190880 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:38.190799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:49:38.194235 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:49:38.194173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6941c1b_fb89_46c2_9c7c_4bd3ff01f02a.slice/crio-8381828f488d8208e5e299a737b9066f759292c77caba839464a1b17ce30b46b WatchSource:0}: Error finding container 8381828f488d8208e5e299a737b9066f759292c77caba839464a1b17ce30b46b: Status 404 returned error can't find the container with id 8381828f488d8208e5e299a737b9066f759292c77caba839464a1b17ce30b46b
Sep 17 17:49:38.712492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:38.712456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" event={"ID":"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a","Type":"ContainerStarted","Data":"8381828f488d8208e5e299a737b9066f759292c77caba839464a1b17ce30b46b"}
Sep 17 17:49:39.718450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:39.718422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" event={"ID":"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a","Type":"ContainerStarted","Data":"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"}
Sep 17 17:49:39.735465 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:39.735418    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" podStartSLOduration=1.285971724 podStartE2EDuration="2.735382603s" totalImagesPullingTime="1.449410879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:39.733589032 +0000 UTC m=+955.392064068" watchObservedRunningTime="2026-09-17 17:49:39.735382603 +0000 UTC m=+955.393857603"
Sep 17 17:49:39.916885 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:39.916824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:47.917019 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:47.916987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:47.929245 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:47.929221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:48.769470 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:48.769441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:49:59.673140 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:49:59.673107    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:50:15.206896 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.206860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:50:15.216020 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:50:15.215975    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:50:15.216020 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:15.216020 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:15.216020 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1" cmd=["/bin/sleep","15"]
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.216041    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" message="" err=<
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:50:15.216053    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerName="main" containerID="cri-o://927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"
Sep 17 17:50:15.216247 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.216100    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerName="main" containerID="cri-o://927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1" gracePeriod=60
Sep 17 17:50:15.557024 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.557000    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:50:15.642500 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvhk\" (UniqueName: \"kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642623 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642623 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642623 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642848 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642848 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home\") pod \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\" (UID: \"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a\") "
Sep 17 17:50:15.642848 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:15.642848 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:15.643055 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.642966    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.643055 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.643003    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.643055 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.643033    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home" (OuterVolumeSpecName: "home") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:15.644693 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.644670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm" (OuterVolumeSpecName: "dshm") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:15.644796 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.644734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:15.644796 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.644762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk" (OuterVolumeSpecName: "kube-api-access-dgvhk") pod "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" (UID: "b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a"). InnerVolumeSpecName "kube-api-access-dgvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:15.744087 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.744031    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvhk\" (UniqueName: \"kubernetes.io/projected/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-kube-api-access-dgvhk\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.744087 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.744054    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.744087 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.744065    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.744087 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.744074    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:50:15.889136 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.889104    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerID="927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1" exitCode=0
Sep 17 17:50:15.889383 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.889178    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"
Sep 17 17:50:15.889383 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.889192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" event={"ID":"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a","Type":"ContainerDied","Data":"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"}
Sep 17 17:50:15.889383 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.889260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf" event={"ID":"b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a","Type":"ContainerDied","Data":"8381828f488d8208e5e299a737b9066f759292c77caba839464a1b17ce30b46b"}
Sep 17 17:50:15.889383 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.889296    2577 scope.go:117] "RemoveContainer" containerID="927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"
Sep 17 17:50:15.898240 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.898220    2577 scope.go:117] "RemoveContainer" containerID="927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"
Sep 17 17:50:15.898532 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:50:15.898514    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1\": container with ID starting with 927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1 not found: ID does not exist" containerID="927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"
Sep 17 17:50:15.898584 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.898541    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1"} err="failed to get container status \"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1\": rpc error: code = NotFound desc = could not find container \"927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1\": container with ID starting with 927382521a74dc96890929671d6d9b9eb962f43cf46b73f4503ac44ef98f8fe1 not found: ID does not exist"
Sep 17 17:50:15.914670 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.914644    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:50:15.919768 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:15.919739    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-m84hf"]
Sep 17 17:50:16.851965 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:16.851932    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" path="/var/lib/kubelet/pods/b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a/volumes"
Sep 17 17:50:18.254882 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:18.254852    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:50:19.672652 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:19.672604    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:29.672182 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:29.672136    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:33.276113 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:33.276072    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" containerID="cri-o://bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240" gracePeriod=45
Sep 17 17:50:39.672347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:39.672307    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:39.672873 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:39.672441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:50:49.672508 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:49.672463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:50:59.672915 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:50:59.672874    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:51:03.527074 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.527051    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:51:03.606911 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.606878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6z6w\" (UniqueName: \"kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w\") pod \"4d53c4d2-f586-4c06-af71-c168e7280c59\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") "
Sep 17 17:51:03.607033 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.606923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs\") pod \"4d53c4d2-f586-4c06-af71-c168e7280c59\" (UID: \"4d53c4d2-f586-4c06-af71-c168e7280c59\") "
Sep 17 17:51:03.608958 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.608935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w" (OuterVolumeSpecName: "kube-api-access-h6z6w") pod "4d53c4d2-f586-4c06-af71-c168e7280c59" (UID: "4d53c4d2-f586-4c06-af71-c168e7280c59"). InnerVolumeSpecName "kube-api-access-h6z6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:03.609034 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.608976    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d53c4d2-f586-4c06-af71-c168e7280c59" (UID: "4d53c4d2-f586-4c06-af71-c168e7280c59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:03.708195 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.708168    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6z6w\" (UniqueName: \"kubernetes.io/projected/4d53c4d2-f586-4c06-af71-c168e7280c59-kube-api-access-h6z6w\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:03.708195 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:03.708192    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d53c4d2-f586-4c06-af71-c168e7280c59-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:04.116354 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.116317    2577 generic.go:356] "Generic (PLEG): container finished" podID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerID="bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240" exitCode=0
Sep 17 17:51:04.116551 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.116363    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"
Sep 17 17:51:04.116551 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.116378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" event={"ID":"4d53c4d2-f586-4c06-af71-c168e7280c59","Type":"ContainerDied","Data":"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"}
Sep 17 17:51:04.116551 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.116446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn" event={"ID":"4d53c4d2-f586-4c06-af71-c168e7280c59","Type":"ContainerDied","Data":"4bd50d0bf1dd529b41dc319f5102b890371b20174541b58fc34d959a076141b7"}
Sep 17 17:51:04.116551 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.116474    2577 scope.go:117] "RemoveContainer" containerID="bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"
Sep 17 17:51:04.125670 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.125645    2577 scope.go:117] "RemoveContainer" containerID="bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"
Sep 17 17:51:04.125961 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:51:04.125936    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240\": container with ID starting with bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240 not found: ID does not exist" containerID="bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"
Sep 17 17:51:04.126044 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.125967    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240"} err="failed to get container status \"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240\": rpc error: code = NotFound desc = could not find container \"bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240\": container with ID starting with bc299f27998083d95666916bc60152b24d19cfb681e504410d642125b2f78240 not found: ID does not exist"
Sep 17 17:51:04.145910 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.145883    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:51:04.154422 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.154386    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-q2fdn"]
Sep 17 17:51:04.851672 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:04.851638    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" path="/var/lib/kubelet/pods/4d53c4d2-f586-4c06-af71-c168e7280c59/volumes"
Sep 17 17:51:12.421917 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.421879    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422213    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main"
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main"
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422235    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerName="main"
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422241    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerName="main"
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422288    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6941c1b-fb89-46c2-9c7c-4bd3ff01f02a" containerName="main"
Sep 17 17:51:12.422302 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.422297    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d53c4d2-f586-4c06-af71-c168e7280c59" containerName="main"
Sep 17 17:51:12.425035 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.425017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.427838 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.427815    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-sxn6v\""
Sep 17 17:51:12.427986 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.427818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:51:12.427986 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.427842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:51:12.427986 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.427855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:51:12.433781 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.433757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:12.472585 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.472585 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw72h\" (UniqueName: \"kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.472832 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.472832 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.472832 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.472949 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.472837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.573802 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.573802 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574076 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574076 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw72h\" (UniqueName: \"kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574076 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574076 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.573902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574371 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.574339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574371 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.574359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.574603 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.574427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.576178 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.576159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.576442 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.576424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.582693 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.582663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw72h\" (UniqueName: \"kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-xlgvq\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:12.735629 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:12.735539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:13.011654 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:13.011630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:13.014248 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:51:13.014215    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26037da6_4b49_44a6_a22b_292cb90d7685.slice/crio-774ea7fef7c7ef32e1de9ce2122abc940d811e9bab0d12b31b475ae299fb9047 WatchSource:0}: Error finding container 774ea7fef7c7ef32e1de9ce2122abc940d811e9bab0d12b31b475ae299fb9047: Status 404 returned error can't find the container with id 774ea7fef7c7ef32e1de9ce2122abc940d811e9bab0d12b31b475ae299fb9047
Sep 17 17:51:13.156123 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:13.156089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" event={"ID":"26037da6-4b49-44a6-a22b-292cb90d7685","Type":"ContainerStarted","Data":"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"}
Sep 17 17:51:13.156267 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:13.156148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" event={"ID":"26037da6-4b49-44a6-a22b-292cb90d7685","Type":"ContainerStarted","Data":"774ea7fef7c7ef32e1de9ce2122abc940d811e9bab0d12b31b475ae299fb9047"}
Sep 17 17:51:13.189949 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:13.189897    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" podStartSLOduration=1.18988078 podStartE2EDuration="1.18988078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:13.188730129 +0000 UTC m=+1048.847205121" watchObservedRunningTime="2026-09-17 17:51:13.18988078 +0000 UTC m=+1048.848355780"
Sep 17 17:51:13.736568 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:13.736524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:22.736188 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:22.736153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:22.748638 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:22.748614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:23.216885 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:23.216855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:51.398076 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.398039    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:51.407647 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:51:51.407612    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:51:51.407647 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:51:51.407647 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:51:51.407647 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551" cmd=["/bin/sleep","15"]
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.407665    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" message="" err=<
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:51:51.407679    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" containerName="main" containerID="cri-o://2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"
Sep 17 17:51:51.407947 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.407713    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" containerName="main" containerID="cri-o://2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551" gracePeriod=60
Sep 17 17:51:51.753274 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.753247    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:51.885143 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885143 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885141    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885431 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885431 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885431 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw72h\" (UniqueName: \"kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885431 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache\") pod \"26037da6-4b49-44a6-a22b-292cb90d7685\" (UID: \"26037da6-4b49-44a6-a22b-292cb90d7685\") "
Sep 17 17:51:51.885632 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home" (OuterVolumeSpecName: "home") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:51.885632 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885551    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:51.885731 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.885654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache" (OuterVolumeSpecName: "model-cache") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:51.887534 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.887507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm" (OuterVolumeSpecName: "dshm") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:51.887641 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.887555    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h" (OuterVolumeSpecName: "kube-api-access-gw72h") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "kube-api-access-gw72h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:51.887641 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.887566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26037da6-4b49-44a6-a22b-292cb90d7685" (UID: "26037da6-4b49-44a6-a22b-292cb90d7685"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986111    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986139    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986147    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26037da6-4b49-44a6-a22b-292cb90d7685-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986173    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986182    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw72h\" (UniqueName: \"kubernetes.io/projected/26037da6-4b49-44a6-a22b-292cb90d7685-kube-api-access-gw72h\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:51.986200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:51.986193    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26037da6-4b49-44a6-a22b-292cb90d7685-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:51:52.335444 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.335334    2577 generic.go:356] "Generic (PLEG): container finished" podID="26037da6-4b49-44a6-a22b-292cb90d7685" containerID="2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551" exitCode=0
Sep 17 17:51:52.335444 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.335377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" event={"ID":"26037da6-4b49-44a6-a22b-292cb90d7685","Type":"ContainerDied","Data":"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"}
Sep 17 17:51:52.335444 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.335384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"
Sep 17 17:51:52.335672 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.335455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq" event={"ID":"26037da6-4b49-44a6-a22b-292cb90d7685","Type":"ContainerDied","Data":"774ea7fef7c7ef32e1de9ce2122abc940d811e9bab0d12b31b475ae299fb9047"}
Sep 17 17:51:52.335672 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.335478    2577 scope.go:117] "RemoveContainer" containerID="2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"
Sep 17 17:51:52.344363 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.344345    2577 scope.go:117] "RemoveContainer" containerID="2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"
Sep 17 17:51:52.344640 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:51:52.344619    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551\": container with ID starting with 2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551 not found: ID does not exist" containerID="2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"
Sep 17 17:51:52.344701 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.344645    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551"} err="failed to get container status \"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551\": rpc error: code = NotFound desc = could not find container \"2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551\": container with ID starting with 2aa614661ff15be3fd130a0cc6401b27e5562a714edbba3be1b6867b7f04a551 not found: ID does not exist"
Sep 17 17:51:52.359742 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.359717    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:52.365106 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.365080    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-xlgvq"]
Sep 17 17:51:52.851185 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:51:52.851154    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" path="/var/lib/kubelet/pods/26037da6-4b49-44a6-a22b-292cb90d7685/volumes"
Sep 17 17:52:50.990193 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.990155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:52:50.990811 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.990651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" containerName="main"
Sep 17 17:52:50.990811 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.990672    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" containerName="main"
Sep 17 17:52:50.990811 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.990761    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="26037da6-4b49-44a6-a22b-292cb90d7685" containerName="main"
Sep 17 17:52:50.994766 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.994742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:50.997421 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.997386    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cm9mx\""
Sep 17 17:52:50.997537 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.997437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 17:52:50.999985 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:50.999955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 17:52:51.000119 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.000092    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 17:52:51.011897 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.011877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:52:51.112244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.112410 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.112410 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.112410 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.112410 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhj7q\" (UniqueName: \"kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.112606 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.112465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213430 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213430 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213677 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhj7q\" (UniqueName: \"kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213677 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213677 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213677 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.213957 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.214070 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.213999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.214114 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.214099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.215694 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.215675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.216255 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.216236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.225455 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.225436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhj7q\" (UniqueName: \"kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.304419 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.304352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:52:51.576940 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.576863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:52:51.579824 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:52:51.579794    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728eb110_4130_4f0e_b521_b21b394b4bdb.slice/crio-3a1b4c57c4156e4d0e2c62a15802191195c971f029adc437f8b4fcaeeef2ae52 WatchSource:0}: Error finding container 3a1b4c57c4156e4d0e2c62a15802191195c971f029adc437f8b4fcaeeef2ae52: Status 404 returned error can't find the container with id 3a1b4c57c4156e4d0e2c62a15802191195c971f029adc437f8b4fcaeeef2ae52
Sep 17 17:52:51.600108 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:51.600083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" event={"ID":"728eb110-4130-4f0e-b521-b21b394b4bdb","Type":"ContainerStarted","Data":"3a1b4c57c4156e4d0e2c62a15802191195c971f029adc437f8b4fcaeeef2ae52"}
Sep 17 17:52:52.605652 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:52.605616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" event={"ID":"728eb110-4130-4f0e-b521-b21b394b4bdb","Type":"ContainerStarted","Data":"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"}
Sep 17 17:52:52.623458 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:52.623414    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" podStartSLOduration=2.623380885 podStartE2EDuration="2.623380885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:52.621603688 +0000 UTC m=+1148.280078700" watchObservedRunningTime="2026-09-17 17:52:52.623380885 +0000 UTC m=+1148.281855885"
Sep 17 17:52:53.304683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:52:53.304650    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:53:01.304512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:01.304466    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:53:01.316968 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:01.316943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:53:01.666988 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:01.666956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:53:27.607253 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.607222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:27.610374 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.610347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.614774 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.614752    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-kpwk6\""
Sep 17 17:53:27.615303 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.615284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 17:53:27.615443 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.615289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 17:53:27.615645 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.615622    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 17:53:27.630307 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.630282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:27.714130 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714275 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.714492 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.714386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815378 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815512 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815766 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815766 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815766 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815921 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815921 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.815921 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.815896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.816059 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.816036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.816286 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.816270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.817778 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.817761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.818016 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.817999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.824103 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.824076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.824313 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.824291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j9tq6\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:27.924234 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:27.924184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:28.198841 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:28.198757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:28.201799 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:53:28.201768    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c4c32a_efe5_4ef8_9e3b_cff29b6cfba1.slice/crio-ca110ab63805da8e589c9a57e3c944ff698ca910758ecf416e4cc6fc7ca1eaa7 WatchSource:0}: Error finding container ca110ab63805da8e589c9a57e3c944ff698ca910758ecf416e4cc6fc7ca1eaa7: Status 404 returned error can't find the container with id ca110ab63805da8e589c9a57e3c944ff698ca910758ecf416e4cc6fc7ca1eaa7
Sep 17 17:53:28.203828 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:28.203809    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:28.789611 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:28.789571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" event={"ID":"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1","Type":"ContainerStarted","Data":"ca110ab63805da8e589c9a57e3c944ff698ca910758ecf416e4cc6fc7ca1eaa7"}
Sep 17 17:53:32.264270 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.264234    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:53:32.264609 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.264308    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:53:32.264609 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.264338    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 17:53:32.810097 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.810065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" event={"ID":"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1","Type":"ContainerStarted","Data":"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"}
Sep 17 17:53:32.829747 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.829687    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" podStartSLOduration=1.769556812 podStartE2EDuration="5.829671032s" totalImagesPullingTime="4.06011422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:32.828454141 +0000 UTC m=+1188.486929141" watchObservedRunningTime="2026-09-17 17:53:32.829671032 +0000 UTC m=+1188.488146045"
Sep 17 17:53:32.925293 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:32.925262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:33.930244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:33.930206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:34.821617 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:34.821589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:34.822760 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:34.822734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:44.820556 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:44.820523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:53:44.821647 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:44.821627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:53:45.240244 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:45.240206    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:45.240511 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:45.240462    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" podUID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" containerName="istio-proxy" containerID="cri-o://5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1" gracePeriod=30
Sep 17 17:53:47.083972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.083947    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:47.165788 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165768    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.165888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.165888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165822    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.165888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.165888 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.166090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.166090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.166090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.165993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.166090 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy\") pod \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\" (UID: \"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1\") "
Sep 17 17:53:47.166445 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.166556 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:47.166746 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data" (OuterVolumeSpecName: "istio-data") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.166846 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.166846 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.166803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.168587 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.168554    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc" (OuterVolumeSpecName: "kube-api-access-z8bjc") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "kube-api-access-z8bjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:47.168742 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.168710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 17:53:47.168844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.168708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token" (OuterVolumeSpecName: "istio-token") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:47.168844 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.168769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" (UID: "40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.267102 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267076    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istiod-ca-cert\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267102 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267099    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-envoy\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267113    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267125    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-credential-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267137    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-podinfo\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267149    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-token\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267162    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-workload-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267174    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-istio-data\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.267261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.267188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8bjc\" (UniqueName: \"kubernetes.io/projected/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1-kube-api-access-z8bjc\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.901829 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.901798    2577 generic.go:356] "Generic (PLEG): container finished" podID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" containerID="5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1" exitCode=0
Sep 17 17:53:47.901991 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.901845    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"
Sep 17 17:53:47.901991 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.901846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" event={"ID":"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1","Type":"ContainerDied","Data":"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"}
Sep 17 17:53:47.901991 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.901950    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6" event={"ID":"40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1","Type":"ContainerDied","Data":"ca110ab63805da8e589c9a57e3c944ff698ca910758ecf416e4cc6fc7ca1eaa7"}
Sep 17 17:53:47.901991 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.901976    2577 scope.go:117] "RemoveContainer" containerID="5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"
Sep 17 17:53:47.911266 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.911249    2577 scope.go:117] "RemoveContainer" containerID="5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"
Sep 17 17:53:47.911536 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:53:47.911519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1\": container with ID starting with 5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1 not found: ID does not exist" containerID="5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"
Sep 17 17:53:47.911587 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.911543    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1"} err="failed to get container status \"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1\": rpc error: code = NotFound desc = could not find container \"5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1\": container with ID starting with 5af1219c1f1fda2c7713c0f87db8bd859b2bad6e17daf4e226cf97ff76fef9a1 not found: ID does not exist"
Sep 17 17:53:47.927617 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.927586    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:47.930375 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:47.930356    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j9tq6"]
Sep 17 17:53:48.522586 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.522558    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:53:48.522948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.522851    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" containerName="istio-proxy"
Sep 17 17:53:48.522948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.522862    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" containerName="istio-proxy"
Sep 17 17:53:48.522948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.522921    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" containerName="istio-proxy"
Sep 17 17:53:48.528584 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.528564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.530971 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.530949    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 17:53:48.531085 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.531047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-s7868\""
Sep 17 17:53:48.531158 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.531119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 17:53:48.531158 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.531140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 17:53:48.531274 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.531253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-hl7p5\""
Sep 17 17:53:48.538683 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.538666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:53:48.577893 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.577864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbt2h\" (UniqueName: \"kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.578001 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.577909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.678902 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.678881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbt2h\" (UniqueName: \"kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.679010 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.678922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.681261 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.681236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.691965 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.691943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbt2h\" (UniqueName: \"kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.841353 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.841298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:48.849854 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:48.849830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1" path="/var/lib/kubelet/pods/40c4c32a-efe5-4ef8-9e3b-cff29b6cfba1/volumes"
Sep 17 17:53:49.116684 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:49.116650    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:53:49.120567 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:53:49.120527    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232fd8d3_346c_46d2_b242_b6c0a809552a.slice/crio-84d562f1d9f347e9f24d26532248d856a3c8893d517db27b749df48fc33f76ae WatchSource:0}: Error finding container 84d562f1d9f347e9f24d26532248d856a3c8893d517db27b749df48fc33f76ae: Status 404 returned error can't find the container with id 84d562f1d9f347e9f24d26532248d856a3c8893d517db27b749df48fc33f76ae
Sep 17 17:53:49.915809 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:49.915767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" event={"ID":"232fd8d3-346c-46d2-b242-b6c0a809552a","Type":"ContainerStarted","Data":"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"}
Sep 17 17:53:49.916157 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:49.915824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" event={"ID":"232fd8d3-346c-46d2-b242-b6c0a809552a","Type":"ContainerStarted","Data":"84d562f1d9f347e9f24d26532248d856a3c8893d517db27b749df48fc33f76ae"}
Sep 17 17:53:49.916157 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:49.915915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:53:49.938109 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:53:49.938059    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podStartSLOduration=1.938045828 podStartE2EDuration="1.938045828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:49.937349823 +0000 UTC m=+1205.595824824" watchObservedRunningTime="2026-09-17 17:53:49.938045828 +0000 UTC m=+1205.596520828"
Sep 17 17:54:20.921116 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:20.921086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:54:38.787703 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:38.787492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:54:38.799467 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:54:38.799422    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:54:38.799467 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.799467 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.799467 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2" cmd=["/bin/sleep","15"]
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:38.799484    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" message="" err=<
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:54:38.799496    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerName="main" containerID="cri-o://6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"
Sep 17 17:54:38.799775 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:38.799542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerName="main" containerID="cri-o://6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2" gracePeriod=60
Sep 17 17:54:39.148173 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.148150    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:54:39.151481 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.151451    2577 generic.go:356] "Generic (PLEG): container finished" podID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerID="6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2" exitCode=0
Sep 17 17:54:39.151582 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.151496    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"
Sep 17 17:54:39.151582 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.151523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" event={"ID":"728eb110-4130-4f0e-b521-b21b394b4bdb","Type":"ContainerDied","Data":"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"}
Sep 17 17:54:39.151582 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.151569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg" event={"ID":"728eb110-4130-4f0e-b521-b21b394b4bdb","Type":"ContainerDied","Data":"3a1b4c57c4156e4d0e2c62a15802191195c971f029adc437f8b4fcaeeef2ae52"}
Sep 17 17:54:39.151698 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.151600    2577 scope.go:117] "RemoveContainer" containerID="6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"
Sep 17 17:54:39.160333 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.160314    2577 scope.go:117] "RemoveContainer" containerID="6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"
Sep 17 17:54:39.160656 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:54:39.160629    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2\": container with ID starting with 6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2 not found: ID does not exist" containerID="6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"
Sep 17 17:54:39.160750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.160658    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2"} err="failed to get container status \"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2\": rpc error: code = NotFound desc = could not find container \"6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2\": container with ID starting with 6e86a3eff3c11ab60be6d3f914cf81fd2618dc9bd59d5ee7f924671e2b1915a2 not found: ID does not exist"
Sep 17 17:54:39.287348 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhj7q\" (UniqueName: \"kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287358    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287380    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287549 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs\") pod \"728eb110-4130-4f0e-b521-b21b394b4bdb\" (UID: \"728eb110-4130-4f0e-b521-b21b394b4bdb\") "
Sep 17 17:54:39.287795 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.287795 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.287795 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.287734    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home" (OuterVolumeSpecName: "home") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.289711 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.289687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q" (OuterVolumeSpecName: "kube-api-access-mhj7q") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "kube-api-access-mhj7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:54:39.289833 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.289717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:54:39.289833 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.289746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm" (OuterVolumeSpecName: "dshm") pod "728eb110-4130-4f0e-b521-b21b394b4bdb" (UID: "728eb110-4130-4f0e-b521-b21b394b4bdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:54:39.388185 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388157    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.388185 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388183    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.388185 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388191    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/728eb110-4130-4f0e-b521-b21b394b4bdb-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.388424 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388201    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhj7q\" (UniqueName: \"kubernetes.io/projected/728eb110-4130-4f0e-b521-b21b394b4bdb-kube-api-access-mhj7q\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.388424 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388210    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.388424 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.388218    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/728eb110-4130-4f0e-b521-b21b394b4bdb-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:54:39.475746 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.475711    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:54:39.484408 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:39.484369    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-ggqdg"]
Sep 17 17:54:40.850849 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:54:40.850816    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" path="/var/lib/kubelet/pods/728eb110-4130-4f0e-b521-b21b394b4bdb/volumes"
Sep 17 17:55:31.658070 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.658040    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:55:31.658601 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.658359    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerName="main"
Sep 17 17:55:31.658601 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.658369    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerName="main"
Sep 17 17:55:31.658601 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.658439    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="728eb110-4130-4f0e-b521-b21b394b4bdb" containerName="main"
Sep 17 17:55:31.661379 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.661358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.664188 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.664158    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 17:55:31.664304 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.664157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 17:55:31.664304 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.664201    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 17:55:31.664304 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.664157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-v9mch\""
Sep 17 17:55:31.673874 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.673853    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:55:31.776721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.776835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.776835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.776835 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.776948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5vf\" (UniqueName: \"kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.776948 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.776904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.877686 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.877664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5vf\" (UniqueName: \"kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.878210 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.878184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.878812 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.878681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.878958 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.878921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.879068 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.878973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.879068 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.879006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.879068 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.879033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.879329 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.879307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.879577 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.879558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.881253 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.881234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.881387 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.881372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.886192 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.886174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5vf\" (UniqueName: \"kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:31.972041 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:31.971988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:32.245469 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:32.245440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:55:32.248042 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:55:32.248017    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464b17e5_22ab_417e_81b1_ed41184f0c0c.slice/crio-47b8954e14fd401bc990ce7c6da74f1a7fd744a2f2dbe6cf6a8ad3e76c981bcf WatchSource:0}: Error finding container 47b8954e14fd401bc990ce7c6da74f1a7fd744a2f2dbe6cf6a8ad3e76c981bcf: Status 404 returned error can't find the container with id 47b8954e14fd401bc990ce7c6da74f1a7fd744a2f2dbe6cf6a8ad3e76c981bcf
Sep 17 17:55:32.402198 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:32.402172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" event={"ID":"464b17e5-22ab-417e-81b1-ed41184f0c0c","Type":"ContainerStarted","Data":"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"}
Sep 17 17:55:32.402335 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:32.402234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" event={"ID":"464b17e5-22ab-417e-81b1-ed41184f0c0c","Type":"ContainerStarted","Data":"47b8954e14fd401bc990ce7c6da74f1a7fd744a2f2dbe6cf6a8ad3e76c981bcf"}
Sep 17 17:55:32.419268 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:32.419221    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" podStartSLOduration=1.4192048179999999 podStartE2EDuration="1.419204818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:32.417773033 +0000 UTC m=+1308.076248033" watchObservedRunningTime="2026-09-17 17:55:32.419204818 +0000 UTC m=+1308.077679819"
Sep 17 17:55:32.972532 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:32.972502    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:41.973117 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:41.973080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:41.986471 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:41.986444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:55:42.456816 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:55:42.456788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:56:06.909194 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:06.909160    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:56:06.918025 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:56:06.917988    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 17:56:06.918025 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.918025 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.918025 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111" cmd=["/bin/sleep","15"]
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:06.918041    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" message="" err=<
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:56:06.918053    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerName="main" containerID="cri-o://45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"
Sep 17 17:56:06.918423 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:06.918110    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerName="main" containerID="cri-o://45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111" gracePeriod=60
Sep 17 17:56:07.260845 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.260822    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:56:07.342581 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.342762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342591    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq5vf\" (UniqueName: \"kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.342762 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.342874 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.342874 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.342874 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache\") pod \"464b17e5-22ab-417e-81b1-ed41184f0c0c\" (UID: \"464b17e5-22ab-417e-81b1-ed41184f0c0c\") "
Sep 17 17:56:07.343018 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.342898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.343096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.343044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home" (OuterVolumeSpecName: "home") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.343166 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.343093    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.343166 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.343106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.344885 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.344856    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:07.344983 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.344882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm" (OuterVolumeSpecName: "dshm") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:07.344983 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.344901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf" (OuterVolumeSpecName: "kube-api-access-cq5vf") pod "464b17e5-22ab-417e-81b1-ed41184f0c0c" (UID: "464b17e5-22ab-417e-81b1-ed41184f0c0c"). InnerVolumeSpecName "kube-api-access-cq5vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:07.444096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.444024    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.444096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.444048    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.444096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.444059    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/464b17e5-22ab-417e-81b1-ed41184f0c0c-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.444096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.444069    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/464b17e5-22ab-417e-81b1-ed41184f0c0c-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.444096 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.444078    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5vf\" (UniqueName: \"kubernetes.io/projected/464b17e5-22ab-417e-81b1-ed41184f0c0c-kube-api-access-cq5vf\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:56:07.565347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.565261    2577 generic.go:356] "Generic (PLEG): container finished" podID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerID="45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111" exitCode=0
Sep 17 17:56:07.565347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.565304    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"
Sep 17 17:56:07.565347 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.565341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" event={"ID":"464b17e5-22ab-417e-81b1-ed41184f0c0c","Type":"ContainerDied","Data":"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"}
Sep 17 17:56:07.565567 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.565385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb" event={"ID":"464b17e5-22ab-417e-81b1-ed41184f0c0c","Type":"ContainerDied","Data":"47b8954e14fd401bc990ce7c6da74f1a7fd744a2f2dbe6cf6a8ad3e76c981bcf"}
Sep 17 17:56:07.565567 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.565423    2577 scope.go:117] "RemoveContainer" containerID="45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"
Sep 17 17:56:07.581276 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.581255    2577 scope.go:117] "RemoveContainer" containerID="45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"
Sep 17 17:56:07.581622 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:56:07.581599    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111\": container with ID starting with 45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111 not found: ID does not exist" containerID="45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"
Sep 17 17:56:07.581697 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.581634    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111"} err="failed to get container status \"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111\": rpc error: code = NotFound desc = could not find container \"45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111\": container with ID starting with 45e2cd25f0076f293e495b22fe72e71ebc085f801b09a4b5e9e95253912e4111 not found: ID does not exist"
Sep 17 17:56:07.592194 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.592168    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:56:07.597840 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:07.597814    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444ts5mb"]
Sep 17 17:56:08.852319 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:08.852287    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" path="/var/lib/kubelet/pods/464b17e5-22ab-417e-81b1-ed41184f0c0c/volumes"
Sep 17 17:56:24.214808 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:24.214776    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:56:30.921546 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:30.921500    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:39.234454 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:39.234416    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main" containerID="cri-o://4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780" gracePeriod=45
Sep 17 17:56:40.921257 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:40.921214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:50.921622 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:50.921585    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:56:50.922007 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:56:50.921677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:57:00.921821 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:00.921780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:57:09.482783 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.482758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:57:09.591850 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.591784    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs\") pod \"232fd8d3-346c-46d2-b242-b6c0a809552a\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") "
Sep 17 17:57:09.591850 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.591830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbt2h\" (UniqueName: \"kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h\") pod \"232fd8d3-346c-46d2-b242-b6c0a809552a\" (UID: \"232fd8d3-346c-46d2-b242-b6c0a809552a\") "
Sep 17 17:57:09.593899 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.593871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h" (OuterVolumeSpecName: "kube-api-access-tbt2h") pod "232fd8d3-346c-46d2-b242-b6c0a809552a" (UID: "232fd8d3-346c-46d2-b242-b6c0a809552a"). InnerVolumeSpecName "kube-api-access-tbt2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:09.594013 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.593918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "232fd8d3-346c-46d2-b242-b6c0a809552a" (UID: "232fd8d3-346c-46d2-b242-b6c0a809552a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:09.692882 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.692848    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232fd8d3-346c-46d2-b242-b6c0a809552a-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:57:09.692882 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.692882    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbt2h\" (UniqueName: \"kubernetes.io/projected/232fd8d3-346c-46d2-b242-b6c0a809552a-kube-api-access-tbt2h\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:57:09.856651 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.856621    2577 generic.go:356] "Generic (PLEG): container finished" podID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerID="4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780" exitCode=0
Sep 17 17:57:09.856750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.856666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"
Sep 17 17:57:09.856750 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.856701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" event={"ID":"232fd8d3-346c-46d2-b242-b6c0a809552a","Type":"ContainerDied","Data":"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"}
Sep 17 17:57:09.856826 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.856751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb" event={"ID":"232fd8d3-346c-46d2-b242-b6c0a809552a","Type":"ContainerDied","Data":"84d562f1d9f347e9f24d26532248d856a3c8893d517db27b749df48fc33f76ae"}
Sep 17 17:57:09.856826 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.856775    2577 scope.go:117] "RemoveContainer" containerID="4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"
Sep 17 17:57:09.865309 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.865287    2577 scope.go:117] "RemoveContainer" containerID="4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"
Sep 17 17:57:09.865625 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:57:09.865606    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780\": container with ID starting with 4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780 not found: ID does not exist" containerID="4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"
Sep 17 17:57:09.865691 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.865632    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780"} err="failed to get container status \"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780\": rpc error: code = NotFound desc = could not find container \"4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780\": container with ID starting with 4c35843b5f6a415077670e6e5eb73f465d2944d45959c879d5bca7c3471bd780 not found: ID does not exist"
Sep 17 17:57:09.878727 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.878704    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:57:09.882229 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:09.882206    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-w6zvb"]
Sep 17 17:57:10.851289 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:10.851258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" path="/var/lib/kubelet/pods/232fd8d3-346c-46d2-b242-b6c0a809552a/volumes"
Sep 17 17:57:20.027697 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.027666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.027996    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main"
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.028008    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main"
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.028014    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerName="main"
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.028019    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerName="main"
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.028074    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="464b17e5-22ab-417e-81b1-ed41184f0c0c" containerName="main"
Sep 17 17:57:20.028100 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.028082    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="232fd8d3-346c-46d2-b242-b6c0a809552a" containerName="main"
Sep 17 17:57:20.030972 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.030951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.033918 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.033897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-bzq6w\""
Sep 17 17:57:20.034058 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.034014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 17:57:20.034124 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.034051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 17:57:20.034200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.034173    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 17:57:20.034335 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.034320    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-86qsh\""
Sep 17 17:57:20.043814 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.043792    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:57:20.168199 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.168171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57fr\" (UniqueName: \"kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.168329 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.168202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.268939 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.268909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57fr\" (UniqueName: \"kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.269081 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.268946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.271378 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.271354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.278793 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.278738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57fr\" (UniqueName: \"kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.341566 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.341533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.621200 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.621172    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:57:20.624134 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:57:20.624108    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dbb410_a139_4689_8084_53e698b7f853.slice/crio-18b7901fa6952c05b7ffaf4c61c917e09bc396d9134497bbabfaff7c671a704a WatchSource:0}: Error finding container 18b7901fa6952c05b7ffaf4c61c917e09bc396d9134497bbabfaff7c671a704a: Status 404 returned error can't find the container with id 18b7901fa6952c05b7ffaf4c61c917e09bc396d9134497bbabfaff7c671a704a
Sep 17 17:57:20.906278 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.906196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" event={"ID":"24dbb410-a139-4689-8084-53e698b7f853","Type":"ContainerStarted","Data":"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"}
Sep 17 17:57:20.906278 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.906253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" event={"ID":"24dbb410-a139-4689-8084-53e698b7f853","Type":"ContainerStarted","Data":"18b7901fa6952c05b7ffaf4c61c917e09bc396d9134497bbabfaff7c671a704a"}
Sep 17 17:57:20.906474 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.906358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:57:20.927950 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:20.927906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podStartSLOduration=0.927891064 podStartE2EDuration="927.891064ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:20.925844685 +0000 UTC m=+1416.584319704" watchObservedRunningTime="2026-09-17 17:57:20.927891064 +0000 UTC m=+1416.586366064"
Sep 17 17:57:38.384923 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.384836    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 17:57:38.388422 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.388386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.390570 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.390546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-7w2kn\""
Sep 17 17:57:38.390734 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.390716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 17:57:38.391579 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.391560    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 17:57:38.391681 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.391580    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-p2vhb\""
Sep 17 17:57:38.391681 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.391631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 17:57:38.396450 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.396423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 17:57:38.400285 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.400265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzztv\" (UniqueName: \"kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.400367 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.400309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.501561 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.501528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.501738 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.501585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzztv\" (UniqueName: \"kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.503990 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.503965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.510675 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.510653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzztv\" (UniqueName: \"kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.700998 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.700907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:38.977556 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.977443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 17:57:38.980422 ip-10-0-128-61 kubenswrapper[2577]: W0917 17:57:38.980378    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ed52ea_e8f4_4cce_9242_0f94378cab2a.slice/crio-77bb3a0fd46feed9c725d0e932a5bc99d7890371a02443d86f8475575632c5eb WatchSource:0}: Error finding container 77bb3a0fd46feed9c725d0e932a5bc99d7890371a02443d86f8475575632c5eb: Status 404 returned error can't find the container with id 77bb3a0fd46feed9c725d0e932a5bc99d7890371a02443d86f8475575632c5eb
Sep 17 17:57:38.990287 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:38.990261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" event={"ID":"c3ed52ea-e8f4-4cce-9242-0f94378cab2a","Type":"ContainerStarted","Data":"77bb3a0fd46feed9c725d0e932a5bc99d7890371a02443d86f8475575632c5eb"}
Sep 17 17:57:39.996871 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:39.996823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" event={"ID":"c3ed52ea-e8f4-4cce-9242-0f94378cab2a","Type":"ContainerStarted","Data":"f804517c7356b381ec11aa3cc3b90bb86a147944b0f69252558ed4fe41c03bdf"}
Sep 17 17:57:39.997277 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:39.996907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:57:40.012668 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:40.012616    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podStartSLOduration=2.01260051 podStartE2EDuration="2.01260051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:40.012507397 +0000 UTC m=+1435.670982397" watchObservedRunningTime="2026-09-17 17:57:40.01260051 +0000 UTC m=+1435.671075512"
Sep 17 17:57:51.912850 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:57:51.912818    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:58:11.003466 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:11.003425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:58:41.913455 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:41.913384    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:41.950296 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:41.950224    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:58:44.847807 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:44.847781    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:58:44.848637 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:44.848610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 17:58:51.913194 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:51.913158    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:56.971487 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:58:56.971428    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" containerID="cri-o://c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a" gracePeriod=45
Sep 17 17:59:01.912727 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:01.912687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:01.913113 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:01.912816    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:59:11.913480 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:11.913379    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:21.912631 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:21.912594    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:27.214583 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.214560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:59:27.319529 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.319472    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b57fr\" (UniqueName: \"kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr\") pod \"24dbb410-a139-4689-8084-53e698b7f853\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") "
Sep 17 17:59:27.319529 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.319514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs\") pod \"24dbb410-a139-4689-8084-53e698b7f853\" (UID: \"24dbb410-a139-4689-8084-53e698b7f853\") "
Sep 17 17:59:27.321674 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.321651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24dbb410-a139-4689-8084-53e698b7f853" (UID: "24dbb410-a139-4689-8084-53e698b7f853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:27.321779 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.321721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr" (OuterVolumeSpecName: "kube-api-access-b57fr") pod "24dbb410-a139-4689-8084-53e698b7f853" (UID: "24dbb410-a139-4689-8084-53e698b7f853"). InnerVolumeSpecName "kube-api-access-b57fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:27.420726 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.420701    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b57fr\" (UniqueName: \"kubernetes.io/projected/24dbb410-a139-4689-8084-53e698b7f853-kube-api-access-b57fr\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:59:27.420726 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.420723    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24dbb410-a139-4689-8084-53e698b7f853-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 17:59:27.485954 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.485925    2577 generic.go:356] "Generic (PLEG): container finished" podID="24dbb410-a139-4689-8084-53e698b7f853" containerID="c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a" exitCode=0
Sep 17 17:59:27.486047 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.485973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"
Sep 17 17:59:27.486047 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.486013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" event={"ID":"24dbb410-a139-4689-8084-53e698b7f853","Type":"ContainerDied","Data":"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"}
Sep 17 17:59:27.486125 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.486070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc" event={"ID":"24dbb410-a139-4689-8084-53e698b7f853","Type":"ContainerDied","Data":"18b7901fa6952c05b7ffaf4c61c917e09bc396d9134497bbabfaff7c671a704a"}
Sep 17 17:59:27.486125 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.486095    2577 scope.go:117] "RemoveContainer" containerID="c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"
Sep 17 17:59:27.494836 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.494819    2577 scope.go:117] "RemoveContainer" containerID="c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"
Sep 17 17:59:27.495143 ip-10-0-128-61 kubenswrapper[2577]: E0917 17:59:27.495124    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a\": container with ID starting with c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a not found: ID does not exist" containerID="c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"
Sep 17 17:59:27.495194 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.495151    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a"} err="failed to get container status \"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a\": rpc error: code = NotFound desc = could not find container \"c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a\": container with ID starting with c80a24802134804f37639615a3e6a1b8c4e3a3879d7163ec9b8206c479ac002a not found: ID does not exist"
Sep 17 17:59:27.512721 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.512696    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:59:27.521801 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:27.521776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69pr5tc"]
Sep 17 17:59:28.856082 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:28.851868    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24dbb410-a139-4689-8084-53e698b7f853" path="/var/lib/kubelet/pods/24dbb410-a139-4689-8084-53e698b7f853/volumes"
Sep 17 17:59:30.501922 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:30.501886    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 17:59:31.002620 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:31.002585    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:41.002779 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:41.002735    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:45.520441 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:45.520383    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" containerID="cri-o://f804517c7356b381ec11aa3cc3b90bb86a147944b0f69252558ed4fe41c03bdf" gracePeriod=45
Sep 17 17:59:51.003257 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:51.003219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:51.003708 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:51.003319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 17:59:57.702574 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.702542    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 17:59:57.703030 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.702881    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main"
Sep 17 17:59:57.703030 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.702892    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main"
Sep 17 17:59:57.703030 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.702960    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24dbb410-a139-4689-8084-53e698b7f853" containerName="main"
Sep 17 17:59:57.705932 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.705911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.708803 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.708777    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-qbq5r\""
Sep 17 17:59:57.708803 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.708794    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 17:59:57.708803 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.708804    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 17:59:57.709010 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.708780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-wltxl\""
Sep 17 17:59:57.709010 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.708790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 17:59:57.714433 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.714408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 17:59:57.837366 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.837331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.837538 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.837423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l47\" (UniqueName: \"kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.938797 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.938758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l47\" (UniqueName: \"kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.938966 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.938823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.941387 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.941367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:57.946782 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:57.946755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l47\" (UniqueName: \"kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:58.017735 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.017662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:58.293664 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.293635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 17:59:58.646093 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.646046    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" event={"ID":"4b5553cd-21eb-4e87-808a-e20cd2d43594","Type":"ContainerStarted","Data":"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"}
Sep 17 17:59:58.646286 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.646143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" event={"ID":"4b5553cd-21eb-4e87-808a-e20cd2d43594","Type":"ContainerStarted","Data":"80ce77f81867126cf476366beb5bb0a059adef8442265b7d8b09f697d95ffed6"}
Sep 17 17:59:58.646286 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.646201    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 17:59:58.668067 ip-10-0-128-61 kubenswrapper[2577]: I0917 17:59:58.668019    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podStartSLOduration=1.668004726 podStartE2EDuration="1.668004726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:58.666581268 +0000 UTC m=+1574.325056269" watchObservedRunningTime="2026-09-17 17:59:58.668004726 +0000 UTC m=+1574.326479759"
Sep 17 18:00:01.003181 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:01.003146    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:11.002658 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:11.002616    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:15.727487 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.727452    2577 generic.go:356] "Generic (PLEG): container finished" podID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerID="f804517c7356b381ec11aa3cc3b90bb86a147944b0f69252558ed4fe41c03bdf" exitCode=0
Sep 17 18:00:15.727833 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.727523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" event={"ID":"c3ed52ea-e8f4-4cce-9242-0f94378cab2a","Type":"ContainerDied","Data":"f804517c7356b381ec11aa3cc3b90bb86a147944b0f69252558ed4fe41c03bdf"}
Sep 17 18:00:15.770776 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.770757    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 18:00:15.873163 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.873139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzztv\" (UniqueName: \"kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv\") pod \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") "
Sep 17 18:00:15.873288 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.873208    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs\") pod \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\" (UID: \"c3ed52ea-e8f4-4cce-9242-0f94378cab2a\") "
Sep 17 18:00:15.875327 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.875304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3ed52ea-e8f4-4cce-9242-0f94378cab2a" (UID: "c3ed52ea-e8f4-4cce-9242-0f94378cab2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:15.875426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.875304    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv" (OuterVolumeSpecName: "kube-api-access-tzztv") pod "c3ed52ea-e8f4-4cce-9242-0f94378cab2a" (UID: "c3ed52ea-e8f4-4cce-9242-0f94378cab2a"). InnerVolumeSpecName "kube-api-access-tzztv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:15.973887 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.973866    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzztv\" (UniqueName: \"kubernetes.io/projected/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-kube-api-access-tzztv\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:00:15.973972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:15.973889    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ed52ea-e8f4-4cce-9242-0f94378cab2a-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:00:16.732612 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.732580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f" event={"ID":"c3ed52ea-e8f4-4cce-9242-0f94378cab2a","Type":"ContainerDied","Data":"77bb3a0fd46feed9c725d0e932a5bc99d7890371a02443d86f8475575632c5eb"}
Sep 17 18:00:16.733045 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.732619    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"
Sep 17 18:00:16.733045 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.732640    2577 scope.go:117] "RemoveContainer" containerID="f804517c7356b381ec11aa3cc3b90bb86a147944b0f69252558ed4fe41c03bdf"
Sep 17 18:00:16.756269 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.756242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 18:00:16.762674 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.762649    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9ctdw9f"]
Sep 17 18:00:16.852902 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:16.852870    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" path="/var/lib/kubelet/pods/c3ed52ea-e8f4-4cce-9242-0f94378cab2a/volumes"
Sep 17 18:00:29.651870 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:29.651836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 18:00:32.914292 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.914256    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:00:32.914685 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.914573    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main"
Sep 17 18:00:32.914685 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.914585    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main"
Sep 17 18:00:32.914685 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.914646    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ed52ea-e8f4-4cce-9242-0f94378cab2a" containerName="main"
Sep 17 18:00:32.919154 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.919127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:32.921369 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.921342    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:00:32.922379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.922351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-b95jf\""
Sep 17 18:00:32.922501 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.922407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:00:32.922501 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.922455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-cpwjn\""
Sep 17 18:00:32.922501 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.922358    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:00:32.926882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.926856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:00:32.994356 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.994329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:32.994528 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:32.994386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6b4\" (UniqueName: \"kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.095007 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.094979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6b4\" (UniqueName: \"kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.095154 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.095027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.097435 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.097404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.103740 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.103718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6b4\" (UniqueName: \"kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.230326 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.230269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.507413 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.507320    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:00:33.510367 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:00:33.510341    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fd25fb_3e5c_4903_a174_1a6b297ed398.slice/crio-423cb9861b5ada36ec5e35f800847579f1b1d2c40cfd778d63662f8149e95b37 WatchSource:0}: Error finding container 423cb9861b5ada36ec5e35f800847579f1b1d2c40cfd778d63662f8149e95b37: Status 404 returned error can't find the container with id 423cb9861b5ada36ec5e35f800847579f1b1d2c40cfd778d63662f8149e95b37
Sep 17 18:00:33.810596 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.810513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" event={"ID":"e8fd25fb-3e5c-4903-a174-1a6b297ed398","Type":"ContainerStarted","Data":"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"}
Sep 17 18:00:33.810596 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.810568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" event={"ID":"e8fd25fb-3e5c-4903-a174-1a6b297ed398","Type":"ContainerStarted","Data":"423cb9861b5ada36ec5e35f800847579f1b1d2c40cfd778d63662f8149e95b37"}
Sep 17 18:00:33.810819 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.810676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:00:33.830617 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:33.830573    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podStartSLOduration=1.8305596149999999 podStartE2EDuration="1.830559615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:33.829379741 +0000 UTC m=+1609.487854743" watchObservedRunningTime="2026-09-17 18:00:33.830559615 +0000 UTC m=+1609.489034614"
Sep 17 18:00:48.217528 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:48.217495    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 18:00:49.651842 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:49.651802    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:59.652476 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:00:59.652433    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:03.236267 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:03.236219    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" containerID="cri-o://f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2" gracePeriod=45
Sep 17 18:01:04.817518 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:04.817485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:01:09.652175 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:09.652133    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:09.652671 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:09.652250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 18:01:19.652178 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:19.652127    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:29.652475 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:29.652427    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:01:33.482677 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.482648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 18:01:33.624847 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.624818    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4l47\" (UniqueName: \"kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47\") pod \"4b5553cd-21eb-4e87-808a-e20cd2d43594\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") "
Sep 17 18:01:33.625005 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.624858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs\") pod \"4b5553cd-21eb-4e87-808a-e20cd2d43594\" (UID: \"4b5553cd-21eb-4e87-808a-e20cd2d43594\") "
Sep 17 18:01:33.627040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.627006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b5553cd-21eb-4e87-808a-e20cd2d43594" (UID: "4b5553cd-21eb-4e87-808a-e20cd2d43594"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:33.627040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.627024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47" (OuterVolumeSpecName: "kube-api-access-d4l47") pod "4b5553cd-21eb-4e87-808a-e20cd2d43594" (UID: "4b5553cd-21eb-4e87-808a-e20cd2d43594"). InnerVolumeSpecName "kube-api-access-d4l47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:33.726317 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.726287    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4l47\" (UniqueName: \"kubernetes.io/projected/4b5553cd-21eb-4e87-808a-e20cd2d43594-kube-api-access-d4l47\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:01:33.726317 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:33.726312    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b5553cd-21eb-4e87-808a-e20cd2d43594-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:01:34.090581 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.090494    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerID="f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2" exitCode=0
Sep 17 18:01:34.090581 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.090536    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"
Sep 17 18:01:34.090581 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.090557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" event={"ID":"4b5553cd-21eb-4e87-808a-e20cd2d43594","Type":"ContainerDied","Data":"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"}
Sep 17 18:01:34.090800 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.090600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts" event={"ID":"4b5553cd-21eb-4e87-808a-e20cd2d43594","Type":"ContainerDied","Data":"80ce77f81867126cf476366beb5bb0a059adef8442265b7d8b09f697d95ffed6"}
Sep 17 18:01:34.090800 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.090621    2577 scope.go:117] "RemoveContainer" containerID="f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"
Sep 17 18:01:34.099405 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.099378    2577 scope.go:117] "RemoveContainer" containerID="f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"
Sep 17 18:01:34.099667 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:01:34.099647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2\": container with ID starting with f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2 not found: ID does not exist" containerID="f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"
Sep 17 18:01:34.099731 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.099675    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2"} err="failed to get container status \"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2\": rpc error: code = NotFound desc = could not find container \"f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2\": container with ID starting with f26ace08dbec679ad43f46b9d7305c626f75c5fd707a2aeefff42812894e27d2 not found: ID does not exist"
Sep 17 18:01:34.114111 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.114088    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 18:01:34.116330 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.116305    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-57859b8ts"]
Sep 17 18:01:34.850736 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:34.850694    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" path="/var/lib/kubelet/pods/4b5553cd-21eb-4e87-808a-e20cd2d43594/volumes"
Sep 17 18:01:55.795054 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.795023    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:01:55.795453 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.795335    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main"
Sep 17 18:01:55.795453 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.795346    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main"
Sep 17 18:01:55.795453 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.795429    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5553cd-21eb-4e87-808a-e20cd2d43594" containerName="main"
Sep 17 18:01:55.798168 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.798146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.800671 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.800646    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:01:55.801121 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.801102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:01:55.801253 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.801223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-twtxs\""
Sep 17 18:01:55.801372 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.801244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-rdtf2\""
Sep 17 18:01:55.801372 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.801251    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:01:55.809176 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.809154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:01:55.887372 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.887344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkpx\" (UniqueName: \"kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.887509 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.887419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.987932 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.987908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.988024 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.987987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkpx\" (UniqueName: \"kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.990438 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.990417    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:55.996028 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:55.996001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkpx\" (UniqueName: \"kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:56.109826 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:56.109799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:56.381229 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:56.381021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:01:56.383957 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:01:56.383927    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d23307a_fd4c_4f87_9fdc_b21e38f742dd.slice/crio-887d1d4988dd745f9bf30c34a0bec68135ff784f7ccda720b46be42a63f8e747 WatchSource:0}: Error finding container 887d1d4988dd745f9bf30c34a0bec68135ff784f7ccda720b46be42a63f8e747: Status 404 returned error can't find the container with id 887d1d4988dd745f9bf30c34a0bec68135ff784f7ccda720b46be42a63f8e747
Sep 17 18:01:57.197039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:57.197000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" event={"ID":"7d23307a-fd4c-4f87-9fdc-b21e38f742dd","Type":"ContainerStarted","Data":"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"}
Sep 17 18:01:57.197455 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:57.197061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" event={"ID":"7d23307a-fd4c-4f87-9fdc-b21e38f742dd","Type":"ContainerStarted","Data":"887d1d4988dd745f9bf30c34a0bec68135ff784f7ccda720b46be42a63f8e747"}
Sep 17 18:01:57.197455 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:57.197156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:01:57.213383 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:01:57.213338    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podStartSLOduration=2.213324537 podStartE2EDuration="2.213324537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:57.212411084 +0000 UTC m=+1692.870886077" watchObservedRunningTime="2026-09-17 18:01:57.213324537 +0000 UTC m=+1692.871799537"
Sep 17 18:02:28.203237 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:28.203209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:02:28.524012 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:28.523935    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:02:33.059562 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:33.059519    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:02:34.816600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:34.816559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:38.203956 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:38.203921    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:43.543599 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:43.543534    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main" containerID="cri-o://cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b" gracePeriod=45
Sep 17 18:02:44.816641 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:44.816602    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:48.080148 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:48.080105    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" containerID="cri-o://21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98" gracePeriod=45
Sep 17 18:02:48.203194 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:48.203160    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:54.817316 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:54.817266    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:54.817719 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:54.817388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:02:58.203346 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:58.203309    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:58.203773 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:02:58.203455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:03:04.816253 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:04.816218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:08.203152 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:08.203108    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:13.785099 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.785073    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:03:13.892029 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.891999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq6b4\" (UniqueName: \"kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4\") pod \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") "
Sep 17 18:03:13.892200 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.892065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs\") pod \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\" (UID: \"e8fd25fb-3e5c-4903-a174-1a6b297ed398\") "
Sep 17 18:03:13.894317 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.894291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8fd25fb-3e5c-4903-a174-1a6b297ed398" (UID: "e8fd25fb-3e5c-4903-a174-1a6b297ed398"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:13.894378 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.894310    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4" (OuterVolumeSpecName: "kube-api-access-qq6b4") pod "e8fd25fb-3e5c-4903-a174-1a6b297ed398" (UID: "e8fd25fb-3e5c-4903-a174-1a6b297ed398"). InnerVolumeSpecName "kube-api-access-qq6b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:13.992851 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.992811    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fd25fb-3e5c-4903-a174-1a6b297ed398-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:03:13.992851 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:13.992844    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq6b4\" (UniqueName: \"kubernetes.io/projected/e8fd25fb-3e5c-4903-a174-1a6b297ed398-kube-api-access-qq6b4\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:03:14.541283 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.541248    2577 generic.go:356] "Generic (PLEG): container finished" podID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerID="cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b" exitCode=0
Sep 17 18:03:14.541500 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.541293    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"
Sep 17 18:03:14.541500 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.541339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" event={"ID":"e8fd25fb-3e5c-4903-a174-1a6b297ed398","Type":"ContainerDied","Data":"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"}
Sep 17 18:03:14.541500 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.541412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6" event={"ID":"e8fd25fb-3e5c-4903-a174-1a6b297ed398","Type":"ContainerDied","Data":"423cb9861b5ada36ec5e35f800847579f1b1d2c40cfd778d63662f8149e95b37"}
Sep 17 18:03:14.541500 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.541437    2577 scope.go:117] "RemoveContainer" containerID="cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"
Sep 17 18:03:14.550046 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.550028    2577 scope.go:117] "RemoveContainer" containerID="cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"
Sep 17 18:03:14.550326 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:03:14.550302    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b\": container with ID starting with cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b not found: ID does not exist" containerID="cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"
Sep 17 18:03:14.550451 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.550330    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b"} err="failed to get container status \"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b\": rpc error: code = NotFound desc = could not find container \"cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b\": container with ID starting with cba225b9991b4868cf3b78a5a715447cbc47d9cb96395179f22980a6a5ee7d0b not found: ID does not exist"
Sep 17 18:03:14.562854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.562833    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:03:14.569618 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.569594    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9htp6"]
Sep 17 18:03:14.850543 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:14.850454    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" path="/var/lib/kubelet/pods/e8fd25fb-3e5c-4903-a174-1a6b297ed398/volumes"
Sep 17 18:03:18.202220 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:03:18.202197    2577 logging.go:55] [core] [Channel #407 SubChannel #408]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.38:9003", ServerName: "10.132.0.38:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.38:9003: connect: connection refused"
Sep 17 18:03:18.333012 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.332982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:03:18.427668 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.427635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnkpx\" (UniqueName: \"kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx\") pod \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") "
Sep 17 18:03:18.427846 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.427687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs\") pod \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\" (UID: \"7d23307a-fd4c-4f87-9fdc-b21e38f742dd\") "
Sep 17 18:03:18.430211 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.430174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx" (OuterVolumeSpecName: "kube-api-access-dnkpx") pod "7d23307a-fd4c-4f87-9fdc-b21e38f742dd" (UID: "7d23307a-fd4c-4f87-9fdc-b21e38f742dd"). InnerVolumeSpecName "kube-api-access-dnkpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:18.430342 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.430202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d23307a-fd4c-4f87-9fdc-b21e38f742dd" (UID: "7d23307a-fd4c-4f87-9fdc-b21e38f742dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:18.529194 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.529161    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkpx\" (UniqueName: \"kubernetes.io/projected/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-kube-api-access-dnkpx\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:03:18.529194 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.529190    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d23307a-fd4c-4f87-9fdc-b21e38f742dd-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:03:18.566447 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.566416    2577 generic.go:356] "Generic (PLEG): container finished" podID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerID="21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98" exitCode=0
Sep 17 18:03:18.566600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.566455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"
Sep 17 18:03:18.566600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.566460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" event={"ID":"7d23307a-fd4c-4f87-9fdc-b21e38f742dd","Type":"ContainerDied","Data":"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"}
Sep 17 18:03:18.566600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.566577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" event={"ID":"7d23307a-fd4c-4f87-9fdc-b21e38f742dd","Type":"ContainerDied","Data":"887d1d4988dd745f9bf30c34a0bec68135ff784f7ccda720b46be42a63f8e747"}
Sep 17 18:03:18.566600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.566601    2577 scope.go:117] "RemoveContainer" containerID="21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"
Sep 17 18:03:18.575765 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.575698    2577 scope.go:117] "RemoveContainer" containerID="21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"
Sep 17 18:03:18.576061 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:03:18.576038    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98\": container with ID starting with 21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98 not found: ID does not exist" containerID="21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"
Sep 17 18:03:18.576130 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.576069    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98"} err="failed to get container status \"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98\": rpc error: code = NotFound desc = could not find container \"21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98\": container with ID starting with 21e113188d27d33f8645275ccd9bde64f2c82c9763f9bbd991a0565cab189d98 not found: ID does not exist"
Sep 17 18:03:18.593252 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.593224    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:03:18.601765 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.601741    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2"]
Sep 17 18:03:18.700977 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.700945    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:03:18.701515 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main"
Sep 17 18:03:18.701515 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701461    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main"
Sep 17 18:03:18.701515 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701480    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main"
Sep 17 18:03:18.701515 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main"
Sep 17 18:03:18.701761 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701583    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fd25fb-3e5c-4903-a174-1a6b297ed398" containerName="main"
Sep 17 18:03:18.701761 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.701602    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main"
Sep 17 18:03:18.706144 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.706121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.709691 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.709668    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:03:18.709691 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.709684    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:03:18.709855 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.709793    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:03:18.710031 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.710010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-pxn27\""
Sep 17 18:03:18.721994 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.721971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:03:18.831868 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.831830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.831882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfbp\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.831918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.831939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.832018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832281 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.832061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832281 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.832093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832281 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.832186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.832438 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.832285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.850842 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.850819    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" path="/var/lib/kubelet/pods/7d23307a-fd4c-4f87-9fdc-b21e38f742dd/volumes"
Sep 17 18:03:18.933511 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfbp\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933511 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.933768 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.933666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.934113 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.934049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.934289 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.934253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.934289 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.934261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.934436 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.934285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.934436 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.934346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.936049 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.936027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.936175 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.936158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.943379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.943348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:18.943501 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:18.943485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfbp\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp\") pod \"router-gateway-1-openshift-default-6cdbb6b55f-nmftv\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:19.017035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.016956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:19.202948 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.202905    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d7pkq2" podUID="7d23307a-fd4c-4f87-9fdc-b21e38f742dd" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.38:9003\" within 1s: context deadline exceeded"
Sep 17 18:03:19.203336 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:03:19.202967    2577 logging.go:55] [core] [Channel #407 SubChannel #408]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.38:9003", ServerName: "10.132.0.38:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.38:9003: operation was canceled"
Sep 17 18:03:19.300795 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.300718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:03:19.304107 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:03:19.304075    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63996720_9dd8_4737_9933_87856c284d02.slice/crio-97fd8ef4a236c884956fc66d21597dc26fd3f9cc4059ec281487971ab7c58445 WatchSource:0}: Error finding container 97fd8ef4a236c884956fc66d21597dc26fd3f9cc4059ec281487971ab7c58445: Status 404 returned error can't find the container with id 97fd8ef4a236c884956fc66d21597dc26fd3f9cc4059ec281487971ab7c58445
Sep 17 18:03:19.306675 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.306638    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:19.306811 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.306716    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:19.306811 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.306765    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:19.580246 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.580143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerStarted","Data":"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"}
Sep 17 18:03:19.580246 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.580232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerStarted","Data":"97fd8ef4a236c884956fc66d21597dc26fd3f9cc4059ec281487971ab7c58445"}
Sep 17 18:03:19.602520 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:19.602470    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podStartSLOduration=1.602452293 podStartE2EDuration="1.602452293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:19.600669484 +0000 UTC m=+1775.259144496" watchObservedRunningTime="2026-09-17 18:03:19.602452293 +0000 UTC m=+1775.260927293"
Sep 17 18:03:20.018239 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:20.018203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:20.022854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:20.022828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:20.586244 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:20.586213    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:20.587326 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:20.587306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:43.698169 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.698142    2577 generic.go:356] "Generic (PLEG): container finished" podID="63996720-9dd8-4737-9933-87856c284d02" containerID="1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220" exitCode=0
Sep 17 18:03:43.698516 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.698220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerDied","Data":"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"}
Sep 17 18:03:43.698603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.698588    2577 scope.go:117] "RemoveContainer" containerID="1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"
Sep 17 18:03:43.699568 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.699542    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:43.699638 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.699606    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:43.699675 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:43.699641    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:03:44.704464 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:44.704431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerStarted","Data":"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"}
Sep 17 18:03:44.881161 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:44.881132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:03:44.881857 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:44.881837    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:03:45.017840 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:45.017809    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:45.018011 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:45.017981    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:46.017807 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:46.017762    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:47.017868 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:47.017823    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:48.017804 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:48.017766    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:49.018052 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:49.018015    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:50.018224 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:50.018180    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 17 18:03:50.586785 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:50.586752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:51.021302 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:51.021261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:51.022155 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:51.022137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:03:59.915899 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.915865    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:03:59.919197 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.919176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:03:59.922628 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.922593    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:03:59.922628 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.922620    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:03:59.922802 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.922601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:03:59.922802 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.922604    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9kf7l\""
Sep 17 18:03:59.936979 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:03:59.936957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:04:00.036651 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.036834 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glwrz\" (UniqueName: \"kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.036834 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.036834 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.036834 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.036989 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.036852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137772 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glwrz\" (UniqueName: \"kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.137972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.137948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.138254 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.138224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.138379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.138356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.138493 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.138377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.140092 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.140069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.140342 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.140326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.148456 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.148429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glwrz\" (UniqueName: \"kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.228930 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.228837    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:00.508766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.506118    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:04:00.511715 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:04:00.511687    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb3d120_16a3_4381_ae1e_5a6b7a2e5b91.slice/crio-d401a56ecbba8b26cfe68f80548f27e9a659a7e51c605cdf9c0fc3e87466fa94 WatchSource:0}: Error finding container d401a56ecbba8b26cfe68f80548f27e9a659a7e51c605cdf9c0fc3e87466fa94: Status 404 returned error can't find the container with id d401a56ecbba8b26cfe68f80548f27e9a659a7e51c605cdf9c0fc3e87466fa94
Sep 17 18:04:00.776039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.775946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" event={"ID":"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91","Type":"ContainerStarted","Data":"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"}
Sep 17 18:04:00.776039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.775997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" event={"ID":"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91","Type":"ContainerStarted","Data":"d401a56ecbba8b26cfe68f80548f27e9a659a7e51c605cdf9c0fc3e87466fa94"}
Sep 17 18:04:00.798753 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:00.798697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" podStartSLOduration=1.798682425 podStartE2EDuration="1.798682425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:00.796854991 +0000 UTC m=+1816.455330026" watchObservedRunningTime="2026-09-17 18:04:00.798682425 +0000 UTC m=+1816.457157425"
Sep 17 18:04:01.229115 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:01.229084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:10.229680 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:10.229639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:10.241717 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:10.241691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:10.838309 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:10.838274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:34.060297 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.060257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:04:34.068997 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:04:34.068959    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:34.068997 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:34.068997 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:34.068997 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed" cmd=["/bin/sleep","15"]
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.069019    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" message="" err=<
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:04:34.069039    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerName="main" containerID="cri-o://a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"
Sep 17 18:04:34.069246 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.069088    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerName="main" containerID="cri-o://a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed" gracePeriod=60
Sep 17 18:04:34.419922 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.419893    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:34.519426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519363    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glwrz\" (UniqueName: \"kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519520    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519567    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm\") pod \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\" (UID: \"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91\") "
Sep 17 18:04:34.519849 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home" (OuterVolumeSpecName: "home") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:34.519849 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:34.519849 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:34.519960 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519878    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.519960 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519897    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.519960 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.519911    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.522010 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.521974    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm" (OuterVolumeSpecName: "dshm") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:34.522162 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.522119    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz" (OuterVolumeSpecName: "kube-api-access-glwrz") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "kube-api-access-glwrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:34.522444 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.522405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" (UID: "6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:34.620643 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.620616    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.620743 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.620647    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glwrz\" (UniqueName: \"kubernetes.io/projected/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-kube-api-access-glwrz\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.620743 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.620659    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:04:34.936626 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.936545    2577 generic.go:356] "Generic (PLEG): container finished" podID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerID="a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed" exitCode=0
Sep 17 18:04:34.936626 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.936589    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"
Sep 17 18:04:34.936816 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.936585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" event={"ID":"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91","Type":"ContainerDied","Data":"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"}
Sep 17 18:04:34.936816 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.936707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l" event={"ID":"6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91","Type":"ContainerDied","Data":"d401a56ecbba8b26cfe68f80548f27e9a659a7e51c605cdf9c0fc3e87466fa94"}
Sep 17 18:04:34.936816 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.936729    2577 scope.go:117] "RemoveContainer" containerID="a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"
Sep 17 18:04:34.945552 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.945529    2577 scope.go:117] "RemoveContainer" containerID="a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"
Sep 17 18:04:34.945851 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:04:34.945816    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed\": container with ID starting with a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed not found: ID does not exist" containerID="a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"
Sep 17 18:04:34.945851 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.945845    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed"} err="failed to get container status \"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed\": rpc error: code = NotFound desc = could not find container \"a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed\": container with ID starting with a8861df83764c9c59028078b3a4dd944fdc06d82dccfc1af55c8b752d737d4ed not found: ID does not exist"
Sep 17 18:04:34.954767 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.954746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:04:34.960839 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:34.960818    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-j2c8l"]
Sep 17 18:04:36.851198 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:04:36.851156    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" path="/var/lib/kubelet/pods/6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91/volumes"
Sep 17 18:05:01.546891 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:01.546853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:05:01.547447 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:01.547053    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" containerID="cri-o://6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e" gracePeriod=30
Sep 17 18:05:04.287974 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.287939    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:05:04.288373 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.288259    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerName="main"
Sep 17 18:05:04.288373 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.288270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerName="main"
Sep 17 18:05:04.288373 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.288328    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb3d120-16a3-4381-ae1e-5a6b7a2e5b91" containerName="main"
Sep 17 18:05:04.291130 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.291109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.294270 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.294242    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:05:04.294380 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.294245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:05:04.294380 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.294303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:05:04.294380 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.294309    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-ch9bb\""
Sep 17 18:05:04.294561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.294313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9vpd5\""
Sep 17 18:05:04.301659 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.301634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:05:04.337000 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.336974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7zs\" (UniqueName: \"kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.337139 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.337011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.437711 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.437675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7zs\" (UniqueName: \"kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.437908 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.437725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.440236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.440207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.448330 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.448308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7zs\" (UniqueName: \"kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.602080 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.602053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:04.882837 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:04.882761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:05:04.886022 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:05:04.885996    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cd635a_b00a_416f_b6f3_b3b7a7386fdf.slice/crio-e4bb9cad83fb3615d6839e9d8afe823eb638a7efc7b4fac6847f9f8579d397f4 WatchSource:0}: Error finding container e4bb9cad83fb3615d6839e9d8afe823eb638a7efc7b4fac6847f9f8579d397f4: Status 404 returned error can't find the container with id e4bb9cad83fb3615d6839e9d8afe823eb638a7efc7b4fac6847f9f8579d397f4
Sep 17 18:05:05.077306 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:05.077262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" event={"ID":"50cd635a-b00a-416f-b6f3-b3b7a7386fdf","Type":"ContainerStarted","Data":"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"}
Sep 17 18:05:05.077516 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:05.077337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" event={"ID":"50cd635a-b00a-416f-b6f3-b3b7a7386fdf","Type":"ContainerStarted","Data":"e4bb9cad83fb3615d6839e9d8afe823eb638a7efc7b4fac6847f9f8579d397f4"}
Sep 17 18:05:05.077516 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:05.077370    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:05:05.099220 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:05.099172    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podStartSLOduration=1.099156671 podStartE2EDuration="1.099156671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:05.097882869 +0000 UTC m=+1880.756357870" watchObservedRunningTime="2026-09-17 18:05:05.099156671 +0000 UTC m=+1880.757631671"
Sep 17 18:05:06.024311 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.024270    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:05:06.806745 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.806719    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:05:06.858049 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858020    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxfbp\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858333    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token\") pod \"63996720-9dd8-4737-9933-87856c284d02\" (UID: \"63996720-9dd8-4737-9933-87856c284d02\") "
Sep 17 18:05:06.858488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858458    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:06.858754 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858507    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:06.858754 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858609    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.858754 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858626    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63996720-9dd8-4737-9933-87856c284d02-istiod-ca-cert\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.858967 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.858905    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:06.859345 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.859292    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:06.859647 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.859615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data" (OuterVolumeSpecName: "istio-data") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:06.860899 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.860865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token" (OuterVolumeSpecName: "istio-token") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:06.861125 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.861097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:05:06.861383 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.861363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:06.861569 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.861550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp" (OuterVolumeSpecName: "kube-api-access-rxfbp") pod "63996720-9dd8-4737-9933-87856c284d02" (UID: "63996720-9dd8-4737-9933-87856c284d02"). InnerVolumeSpecName "kube-api-access-rxfbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:06.960009 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.959974    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-envoy\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960009 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960003    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-workload-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960009 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960012    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-istio-data\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960222 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960020    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-istio-token\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960222 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960029    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfbp\" (UniqueName: \"kubernetes.io/projected/63996720-9dd8-4737-9933-87856c284d02-kube-api-access-rxfbp\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960222 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960037    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63996720-9dd8-4737-9933-87856c284d02-istio-podinfo\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:06.960222 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:06.960047    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63996720-9dd8-4737-9933-87856c284d02-credential-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:05:07.088155 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.088121    2577 generic.go:356] "Generic (PLEG): container finished" podID="63996720-9dd8-4737-9933-87856c284d02" containerID="6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e" exitCode=0
Sep 17 18:05:07.088603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.088168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerDied","Data":"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"}
Sep 17 18:05:07.088603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.088174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"
Sep 17 18:05:07.088603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.088211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv" event={"ID":"63996720-9dd8-4737-9933-87856c284d02","Type":"ContainerDied","Data":"97fd8ef4a236c884956fc66d21597dc26fd3f9cc4059ec281487971ab7c58445"}
Sep 17 18:05:07.088603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.088233    2577 scope.go:117] "RemoveContainer" containerID="6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"
Sep 17 18:05:07.098178 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.098158    2577 scope.go:117] "RemoveContainer" containerID="1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"
Sep 17 18:05:07.105737 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.105712    2577 scope.go:117] "RemoveContainer" containerID="6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"
Sep 17 18:05:07.106011 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:05:07.105990    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e\": container with ID starting with 6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e not found: ID does not exist" containerID="6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"
Sep 17 18:05:07.106089 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.106020    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e"} err="failed to get container status \"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e\": rpc error: code = NotFound desc = could not find container \"6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e\": container with ID starting with 6ebd949e4265b573a2c02fbc2924c88167b052a690110b2ea0983b88bd9ec98e not found: ID does not exist"
Sep 17 18:05:07.106089 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.106045    2577 scope.go:117] "RemoveContainer" containerID="1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"
Sep 17 18:05:07.106317 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:05:07.106296    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220\": container with ID starting with 1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220 not found: ID does not exist" containerID="1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"
Sep 17 18:05:07.106373 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.106327    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220"} err="failed to get container status \"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220\": rpc error: code = NotFound desc = could not find container \"1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220\": container with ID starting with 1d113347c85727c938545024e6948a20d9eaccd48984b348b6309b5e30d5b220 not found: ID does not exist"
Sep 17 18:05:07.117798 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.117769    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:05:07.123300 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:07.123279    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6cdbb6b55f-nmftv"]
Sep 17 18:05:08.850388 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:08.850350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63996720-9dd8-4737-9933-87856c284d02" path="/var/lib/kubelet/pods/63996720-9dd8-4737-9933-87856c284d02/volumes"
Sep 17 18:05:36.083456 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:05:36.083425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:06:11.323194 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323495    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323519    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323525    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323573    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.323766 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.323584    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="63996720-9dd8-4737-9933-87856c284d02" containerName="istio-proxy"
Sep 17 18:06:11.326818 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.326792    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.331292 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.331263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cj5jk\""
Sep 17 18:06:11.331292 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.331283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:06:11.331714 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.331690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:06:11.331839 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.331825    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:06:11.338156 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.338130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:06:11.432272 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.432476 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxk6k\" (UniqueName: \"kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.432476 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.432476 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.432476 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.432631 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.432493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533152 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533344 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533344 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533344 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxk6k\" (UniqueName: \"kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533727 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.533966 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.534099 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.533981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.535705 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.535680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.536488 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.536453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.542222 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.542195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxk6k\" (UniqueName: \"kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.641033 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.641004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:11.911892 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:11.911866    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:06:11.914408 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:06:11.914369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bce32e_d6fe_44df_a250_f371f47cce70.slice/crio-cf92d8e182f407dc26bf43c7631548c923f1a95e5df35038e5c1a80f1b0b37e9 WatchSource:0}: Error finding container cf92d8e182f407dc26bf43c7631548c923f1a95e5df35038e5c1a80f1b0b37e9: Status 404 returned error can't find the container with id cf92d8e182f407dc26bf43c7631548c923f1a95e5df35038e5c1a80f1b0b37e9
Sep 17 18:06:12.386764 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:12.386729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" event={"ID":"76bce32e-d6fe-44df-a250-f371f47cce70","Type":"ContainerStarted","Data":"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"}
Sep 17 18:06:12.387173 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:12.386780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" event={"ID":"76bce32e-d6fe-44df-a250-f371f47cce70","Type":"ContainerStarted","Data":"cf92d8e182f407dc26bf43c7631548c923f1a95e5df35038e5c1a80f1b0b37e9"}
Sep 17 18:06:12.410117 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:12.410063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" podStartSLOduration=1.410047466 podStartE2EDuration="1.410047466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:12.408074662 +0000 UTC m=+1948.066549664" watchObservedRunningTime="2026-09-17 18:06:12.410047466 +0000 UTC m=+1948.068522487"
Sep 17 18:06:12.642224 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:12.642141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:21.641928 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:21.641893    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:21.654198 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:21.654171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:22.443377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:22.443345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:06:54.464320 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.464243    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:06:54.467758 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.467738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.470280 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.470253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-qft44\""
Sep 17 18:06:54.470528 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.470251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:06:54.476657 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.476626    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:06:54.568678 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.568646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.568840 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.568699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqk85\" (UniqueName: \"kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.669380 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.669344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.669578 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.669414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqk85\" (UniqueName: \"kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.671845 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.671817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.685993 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.685966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqk85\" (UniqueName: \"kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:54.779719 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:54.779635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:55.057578 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:55.057435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:06:55.059903 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:06:55.059877    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd26bdcf_8778_4798_b93a_cdc87d3dd432.slice/crio-169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da WatchSource:0}: Error finding container 169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da: Status 404 returned error can't find the container with id 169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da
Sep 17 18:06:55.589613 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:55.589575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" event={"ID":"bd26bdcf-8778-4798-b93a-cdc87d3dd432","Type":"ContainerStarted","Data":"5e177744e5e0f2dbc58d4f06f58b3a7e5849419a9c7ea7e30d6327bf49ffd68c"}
Sep 17 18:06:55.589613 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:55.589629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" event={"ID":"bd26bdcf-8778-4798-b93a-cdc87d3dd432","Type":"ContainerStarted","Data":"169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da"}
Sep 17 18:06:55.590115 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:55.589670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:06:55.616478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:06:55.616419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podStartSLOduration=1.616380336 podStartE2EDuration="1.616380336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:55.615701693 +0000 UTC m=+1991.274176695" watchObservedRunningTime="2026-09-17 18:06:55.616380336 +0000 UTC m=+1991.274855335"
Sep 17 18:07:26.595861 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:26.595825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:07:28.684410 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:28.684367    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:07:36.596184 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:36.596145    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:38.980924 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:38.980890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:07:43.704824 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:43.704760    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main" containerID="cri-o://5e177744e5e0f2dbc58d4f06f58b3a7e5849419a9c7ea7e30d6327bf49ffd68c" gracePeriod=45
Sep 17 18:07:46.083674 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:46.083629    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:46.596109 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:46.596070    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:53.999770 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:53.999703    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main" containerID="cri-o://e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3" gracePeriod=45
Sep 17 18:07:56.083156 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:56.083118    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:56.596064 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:56.596024    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:56.596246 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:07:56.596135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:08:06.083706 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:06.083664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:06.084190 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:06.083769    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:08:06.596011 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:06.595976    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:13.948069 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:13.948040    2577 generic.go:356] "Generic (PLEG): container finished" podID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerID="5e177744e5e0f2dbc58d4f06f58b3a7e5849419a9c7ea7e30d6327bf49ffd68c" exitCode=0
Sep 17 18:08:13.948420 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:13.948114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" event={"ID":"bd26bdcf-8778-4798-b93a-cdc87d3dd432","Type":"ContainerDied","Data":"5e177744e5e0f2dbc58d4f06f58b3a7e5849419a9c7ea7e30d6327bf49ffd68c"}
Sep 17 18:08:13.948420 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:13.948168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z" event={"ID":"bd26bdcf-8778-4798-b93a-cdc87d3dd432","Type":"ContainerDied","Data":"169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da"}
Sep 17 18:08:13.948420 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:13.948186    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="169ec7b9c4a1d0b27cd7ada6ddb2985b9780b41f1cd2cad2c6e9ba6adafbc7da"
Sep 17 18:08:13.949680 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:13.949665    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:08:14.066182 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.066095    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqk85\" (UniqueName: \"kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85\") pod \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") "
Sep 17 18:08:14.066182 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.066137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs\") pod \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\" (UID: \"bd26bdcf-8778-4798-b93a-cdc87d3dd432\") "
Sep 17 18:08:14.068375 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.068347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd26bdcf-8778-4798-b93a-cdc87d3dd432" (UID: "bd26bdcf-8778-4798-b93a-cdc87d3dd432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:14.068508 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.068421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85" (OuterVolumeSpecName: "kube-api-access-bqk85") pod "bd26bdcf-8778-4798-b93a-cdc87d3dd432" (UID: "bd26bdcf-8778-4798-b93a-cdc87d3dd432"). InnerVolumeSpecName "kube-api-access-bqk85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:14.167254 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.167217    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqk85\" (UniqueName: \"kubernetes.io/projected/bd26bdcf-8778-4798-b93a-cdc87d3dd432-kube-api-access-bqk85\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:14.167254 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.167246    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd26bdcf-8778-4798-b93a-cdc87d3dd432-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:14.953088 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.953049    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"
Sep 17 18:08:14.973638 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.973603    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:08:14.984296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:14.984265    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schegbj2z"]
Sep 17 18:08:16.083419 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:16.083366    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:08:16.856174 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:16.850804    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" path="/var/lib/kubelet/pods/bd26bdcf-8778-4798-b93a-cdc87d3dd432/volumes"
Sep 17 18:08:19.016980 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.016947    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:08:19.025764 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:08:19.025721    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:08:19.025764 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:19.025764 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:19.025764 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965" cmd=["/bin/sleep","15"]
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.025777    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" message="" err=<
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:08:19.025790    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" containerName="main" containerID="cri-o://ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"
Sep 17 18:08:19.026003 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.025823    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" containerName="main" containerID="cri-o://ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965" gracePeriod=60
Sep 17 18:08:19.376555 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.376532    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:08:19.510430 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxk6k\" (UniqueName: \"kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510499    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510798 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache\") pod \"76bce32e-d6fe-44df-a250-f371f47cce70\" (UID: \"76bce32e-d6fe-44df-a250-f371f47cce70\") "
Sep 17 18:08:19.510798 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home" (OuterVolumeSpecName: "home") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:19.510918 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:19.510918 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.510877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache" (OuterVolumeSpecName: "model-cache") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:19.512584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.512561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:19.512697 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.512615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm" (OuterVolumeSpecName: "dshm") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:19.512697 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.512676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k" (OuterVolumeSpecName: "kube-api-access-wxk6k") pod "76bce32e-d6fe-44df-a250-f371f47cce70" (UID: "76bce32e-d6fe-44df-a250-f371f47cce70"). InnerVolumeSpecName "kube-api-access-wxk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:19.611251 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611231    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.611251 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611250    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.611377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611259    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.611377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611268    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76bce32e-d6fe-44df-a250-f371f47cce70-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.611377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611277    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76bce32e-d6fe-44df-a250-f371f47cce70-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.611377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.611285    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxk6k\" (UniqueName: \"kubernetes.io/projected/76bce32e-d6fe-44df-a250-f371f47cce70-kube-api-access-wxk6k\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:19.977996 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.977910    2577 generic.go:356] "Generic (PLEG): container finished" podID="76bce32e-d6fe-44df-a250-f371f47cce70" containerID="ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965" exitCode=0
Sep 17 18:08:19.977996 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.977954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"
Sep 17 18:08:19.977996 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.977975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" event={"ID":"76bce32e-d6fe-44df-a250-f371f47cce70","Type":"ContainerDied","Data":"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"}
Sep 17 18:08:19.978260 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.978027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt" event={"ID":"76bce32e-d6fe-44df-a250-f371f47cce70","Type":"ContainerDied","Data":"cf92d8e182f407dc26bf43c7631548c923f1a95e5df35038e5c1a80f1b0b37e9"}
Sep 17 18:08:19.978260 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.978054    2577 scope.go:117] "RemoveContainer" containerID="ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"
Sep 17 18:08:19.988853 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.988831    2577 scope.go:117] "RemoveContainer" containerID="ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"
Sep 17 18:08:19.989275 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:08:19.989257    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965\": container with ID starting with ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965 not found: ID does not exist" containerID="ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"
Sep 17 18:08:19.989347 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:19.989290    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965"} err="failed to get container status \"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965\": rpc error: code = NotFound desc = could not find container \"ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965\": container with ID starting with ce881ae2e6ec395342b1d35cd1fdcd9ead868448dae64d6a71fd214e77cf6965 not found: ID does not exist"
Sep 17 18:08:20.014603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:20.014580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:08:20.025267 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:20.025244    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-hlbrt"]
Sep 17 18:08:20.850965 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:20.850931    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" path="/var/lib/kubelet/pods/76bce32e-d6fe-44df-a250-f371f47cce70/volumes"
Sep 17 18:08:24.251490 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.251463    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:08:24.348635 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.348559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l7zs\" (UniqueName: \"kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs\") pod \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") "
Sep 17 18:08:24.348635 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.348628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs\") pod \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\" (UID: \"50cd635a-b00a-416f-b6f3-b3b7a7386fdf\") "
Sep 17 18:08:24.350793 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.350758    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50cd635a-b00a-416f-b6f3-b3b7a7386fdf" (UID: "50cd635a-b00a-416f-b6f3-b3b7a7386fdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:24.350793 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.350776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs" (OuterVolumeSpecName: "kube-api-access-8l7zs") pod "50cd635a-b00a-416f-b6f3-b3b7a7386fdf" (UID: "50cd635a-b00a-416f-b6f3-b3b7a7386fdf"). InnerVolumeSpecName "kube-api-access-8l7zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:24.449323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.449290    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:24.449323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:24.449319    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l7zs\" (UniqueName: \"kubernetes.io/projected/50cd635a-b00a-416f-b6f3-b3b7a7386fdf-kube-api-access-8l7zs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:08:25.007654 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.007621    2577 generic.go:356] "Generic (PLEG): container finished" podID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerID="e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3" exitCode=0
Sep 17 18:08:25.007865 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.007673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"
Sep 17 18:08:25.007865 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.007704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" event={"ID":"50cd635a-b00a-416f-b6f3-b3b7a7386fdf","Type":"ContainerDied","Data":"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"}
Sep 17 18:08:25.007865 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.007762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2" event={"ID":"50cd635a-b00a-416f-b6f3-b3b7a7386fdf","Type":"ContainerDied","Data":"e4bb9cad83fb3615d6839e9d8afe823eb638a7efc7b4fac6847f9f8579d397f4"}
Sep 17 18:08:25.007865 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.007786    2577 scope.go:117] "RemoveContainer" containerID="e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"
Sep 17 18:08:25.016051 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.016034    2577 scope.go:117] "RemoveContainer" containerID="e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"
Sep 17 18:08:25.016308 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:08:25.016289    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3\": container with ID starting with e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3 not found: ID does not exist" containerID="e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"
Sep 17 18:08:25.016357 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.016321    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3"} err="failed to get container status \"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3\": rpc error: code = NotFound desc = could not find container \"e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3\": container with ID starting with e16864d276f34bc92769abef91426d9ac5d57c7c2b56fe65f95d675a4b783bc3 not found: ID does not exist"
Sep 17 18:08:25.027423 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.027370    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:08:25.034442 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:25.034418    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhmr2"]
Sep 17 18:08:26.850663 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:26.850629    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" path="/var/lib/kubelet/pods/50cd635a-b00a-416f-b6f3-b3b7a7386fdf/volumes"
Sep 17 18:08:32.627846 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.627800    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:08:32.628271 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628254    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" containerName="main"
Sep 17 18:08:32.628316 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" containerName="main"
Sep 17 18:08:32.628316 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628288    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main"
Sep 17 18:08:32.628316 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628297    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main"
Sep 17 18:08:32.628426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628325    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main"
Sep 17 18:08:32.628426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main"
Sep 17 18:08:32.628502 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628425    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd26bdcf-8778-4798-b93a-cdc87d3dd432" containerName="main"
Sep 17 18:08:32.628502 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628439    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="76bce32e-d6fe-44df-a250-f371f47cce70" containerName="main"
Sep 17 18:08:32.628502 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.628453    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cd635a-b00a-416f-b6f3-b3b7a7386fdf" containerName="main"
Sep 17 18:08:32.633379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.633358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.635780 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.635753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:08:32.635890 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.635823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-k5jlq\""
Sep 17 18:08:32.635890 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.635853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:08:32.636007 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.635903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:08:32.643457 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.643435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:08:32.813686 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813686 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwnxh\" (UniqueName: \"kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.813882 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.813855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.914781 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwnxh\" (UniqueName: \"kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.914781 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.914781 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.914781 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915086 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915086 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915086 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.914895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915190 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.915148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915269 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.915245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.915277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.915323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.915315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.917020 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.916997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.917279 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.917261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.925726 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.925706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwnxh\" (UniqueName: \"kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:32.943525 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:32.943502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:08:33.229033 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:33.228952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:08:33.266203 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:33.266186    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:34.051777 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:34.051739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerStarted","Data":"864b09943e673c1495b8b0dcdb9831f6c90960f199cbd7335e8066386e1f5292"}
Sep 17 18:08:42.097990 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:42.097895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerStarted","Data":"16adb99f30c96d576e5fb63a3d41b5b8d2f350d0ca7484788b6ee522508c8c22"}
Sep 17 18:08:43.103861 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:43.103828    2577 generic.go:356] "Generic (PLEG): container finished" podID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerID="16adb99f30c96d576e5fb63a3d41b5b8d2f350d0ca7484788b6ee522508c8c22" exitCode=0
Sep 17 18:08:43.104231 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:43.103918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerDied","Data":"16adb99f30c96d576e5fb63a3d41b5b8d2f350d0ca7484788b6ee522508c8c22"}
Sep 17 18:08:45.565651 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:45.565621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:08:45.566131 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:08:45.565684    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:09:15.280299 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:15.280269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerStarted","Data":"d6909b5b00897bedfb0b9746ffdc69a5196af1857d58e2e7f5ce370d4759e81f"}
Sep 17 18:09:16.288074 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:16.288041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerStarted","Data":"b0b978d91b7ba4b5687c2b2828b8b30bf9544d2048cbe2d0725190ea7b13f4d7"}
Sep 17 18:09:16.311724 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:16.311668    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podStartSLOduration=3.254170481 podStartE2EDuration="44.311653332s" totalImagesPullingTime="39.971090375s" totalInitContainerRuntime="1.086392476s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:16.31072623 +0000 UTC m=+2131.969201248" watchObservedRunningTime="2026-09-17 18:09:16.311653332 +0000 UTC m=+2131.970128331"
Sep 17 18:09:16.943657 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:16.943625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:09:22.944719 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.944676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:09:22.946265 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.946232    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:09:22.956602 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.956572    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:09:22.958864 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.958843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:22.961991 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.961967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-tbq76\""
Sep 17 18:09:22.962114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.961970    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:09:22.962114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.961970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-qwq9g\""
Sep 17 18:09:22.962114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.961991    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:09:22.962294 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.962008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:09:22.970935 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:22.970907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:09:23.130086 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.130053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6gd\" (UniqueName: \"kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.130269 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.130127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.230773 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.230674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6gd\" (UniqueName: \"kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.230958 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.230793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.233417 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.233362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.242385 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.242358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6gd\" (UniqueName: \"kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.272007 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.271970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:23.550943 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:23.550916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:09:23.553326 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:09:23.553285    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6888c6c4_2f19_456d_b9fc_349abb6a385d.slice/crio-09f3f04efd77a97e652517d4e29997275e3cac7986af5c4a9801ccd2d009bb7d WatchSource:0}: Error finding container 09f3f04efd77a97e652517d4e29997275e3cac7986af5c4a9801ccd2d009bb7d: Status 404 returned error can't find the container with id 09f3f04efd77a97e652517d4e29997275e3cac7986af5c4a9801ccd2d009bb7d
Sep 17 18:09:24.330934 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:24.330901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" event={"ID":"6888c6c4-2f19-456d-b9fc-349abb6a385d","Type":"ContainerStarted","Data":"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"}
Sep 17 18:09:24.331345 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:24.330951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" event={"ID":"6888c6c4-2f19-456d-b9fc-349abb6a385d","Type":"ContainerStarted","Data":"09f3f04efd77a97e652517d4e29997275e3cac7986af5c4a9801ccd2d009bb7d"}
Sep 17 18:09:24.331345 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:24.331003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:09:24.349721 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:24.349672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podStartSLOduration=2.349655128 podStartE2EDuration="2.349655128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:24.347883612 +0000 UTC m=+2140.006358613" watchObservedRunningTime="2026-09-17 18:09:24.349655128 +0000 UTC m=+2140.008130127"
Sep 17 18:09:32.944233 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:32.944172    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:09:42.944382 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:42.944285    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:09:52.944377 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:52.944333    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:09:55.338109 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:09:55.338078    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:10:02.944167 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:02.944126    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:12.944092 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:12.944054    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:15.406779 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.406743    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:10:15.409452 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.409432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.413363 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.413332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-kflr5\""
Sep 17 18:10:15.413537 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.413379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:10:15.430816 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.430787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:10:15.561024 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.560982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5n9\" (UniqueName: \"kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.561024 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.561027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.662277 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.662173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.662277 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.662261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5n9\" (UniqueName: \"kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.664748 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.664724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.675603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.675578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5n9\" (UniqueName: \"kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:15.720254 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:15.720215    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:16.001521 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:16.001485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:10:16.004371 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:10:16.004343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708095bb_43a1_4db6_afd2_0b37eb06f2c3.slice/crio-67ef45208ee0bf3262602066ee2d326d8d89ab25a3307cf2b0f752d9816960d3 WatchSource:0}: Error finding container 67ef45208ee0bf3262602066ee2d326d8d89ab25a3307cf2b0f752d9816960d3: Status 404 returned error can't find the container with id 67ef45208ee0bf3262602066ee2d326d8d89ab25a3307cf2b0f752d9816960d3
Sep 17 18:10:16.590126 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:16.590091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" event={"ID":"708095bb-43a1-4db6-afd2-0b37eb06f2c3","Type":"ContainerStarted","Data":"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"}
Sep 17 18:10:16.590581 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:16.590148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" event={"ID":"708095bb-43a1-4db6-afd2-0b37eb06f2c3","Type":"ContainerStarted","Data":"67ef45208ee0bf3262602066ee2d326d8d89ab25a3307cf2b0f752d9816960d3"}
Sep 17 18:10:16.590581 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:16.590196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:16.610305 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:16.610253    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podStartSLOduration=1.610240071 podStartE2EDuration="1.610240071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:16.608207941 +0000 UTC m=+2192.266682940" watchObservedRunningTime="2026-09-17 18:10:16.610240071 +0000 UTC m=+2192.268715102"
Sep 17 18:10:22.944205 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:22.944158    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:32.944261 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:32.944210    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:42.944006 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:42.943946    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:47.596624 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:47.596548    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:10:49.684536 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:49.684496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:10:52.944732 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:52.944687    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:10:57.596797 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:10:57.596748    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:02.944930 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:02.944878    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Sep 17 18:11:04.704013 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:04.703971    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main" containerID="cri-o://4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f" gracePeriod=45
Sep 17 18:11:07.596351 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:07.596269    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:12.953866 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:12.953829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:11:12.961614 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:12.961587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:11:17.596872 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:17.596823    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:17.597282 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:17.596964    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:11:18.114506 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:18.114466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:11:18.114840 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:18.114806    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar" containerID="cri-o://b0b978d91b7ba4b5687c2b2828b8b30bf9544d2048cbe2d0725190ea7b13f4d7" gracePeriod=60
Sep 17 18:11:18.907003 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:18.906975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p_33c5a4d1-0c39-4b89-bb27-4813375f6b5b/modelcar/0.log"
Sep 17 18:11:18.907750 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:18.907657    2577 generic.go:356] "Generic (PLEG): container finished" podID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerID="b0b978d91b7ba4b5687c2b2828b8b30bf9544d2048cbe2d0725190ea7b13f4d7" exitCode=143
Sep 17 18:11:18.907750 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:18.907710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerDied","Data":"b0b978d91b7ba4b5687c2b2828b8b30bf9544d2048cbe2d0725190ea7b13f4d7"}
Sep 17 18:11:27.596731 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:27.596692    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:33.137701 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.137657    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" containerID="cri-o://d6909b5b00897bedfb0b9746ffdc69a5196af1857d58e2e7f5ce370d4759e81f" gracePeriod=45
Sep 17 18:11:33.976051 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.976025    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p_33c5a4d1-0c39-4b89-bb27-4813375f6b5b/modelcar/0.log"
Sep 17 18:11:33.976784 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.976759    2577 generic.go:356] "Generic (PLEG): container finished" podID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerID="d6909b5b00897bedfb0b9746ffdc69a5196af1857d58e2e7f5ce370d4759e81f" exitCode=0
Sep 17 18:11:33.976906 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.976802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerDied","Data":"d6909b5b00897bedfb0b9746ffdc69a5196af1857d58e2e7f5ce370d4759e81f"}
Sep 17 18:11:33.976906 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.976863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" event={"ID":"33c5a4d1-0c39-4b89-bb27-4813375f6b5b","Type":"ContainerDied","Data":"864b09943e673c1495b8b0dcdb9831f6c90960f199cbd7335e8066386e1f5292"}
Sep 17 18:11:33.976906 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.976884    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864b09943e673c1495b8b0dcdb9831f6c90960f199cbd7335e8066386e1f5292"
Sep 17 18:11:33.991161 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.991142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p_33c5a4d1-0c39-4b89-bb27-4813375f6b5b/modelcar/0.log"
Sep 17 18:11:33.994125 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:33.994109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:11:34.023408 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023356    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023429    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwnxh\" (UniqueName: \"kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023803 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.023803 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.023779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:34.024034 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.024012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:34.025738 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.025708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:34.025896 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.025874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm" (OuterVolumeSpecName: "dshm") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:34.026178 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.026152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh" (OuterVolumeSpecName: "kube-api-access-mwnxh") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "kube-api-access-mwnxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:34.038274 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.038246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:34.124132 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home\") pod \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\" (UID: \"33c5a4d1-0c39-4b89-bb27-4813375f6b5b\") "
Sep 17 18:11:34.124269 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124255    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124271    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124281    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kserve-provision-location\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124289    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124297    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124304    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwnxh\" (UniqueName: \"kubernetes.io/projected/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-kube-api-access-mwnxh\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.124587 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.124567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home" (OuterVolumeSpecName: "home") pod "33c5a4d1-0c39-4b89-bb27-4813375f6b5b" (UID: "33c5a4d1-0c39-4b89-bb27-4813375f6b5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:34.224679 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.224654    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c5a4d1-0c39-4b89-bb27-4813375f6b5b-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:34.945386 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.945362    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:11:34.955000 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.954961    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:11:34.982941 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.982878    2577 generic.go:356] "Generic (PLEG): container finished" podID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerID="4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f" exitCode=0
Sep 17 18:11:34.983039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.982936    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"
Sep 17 18:11:34.983039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.982966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" event={"ID":"708095bb-43a1-4db6-afd2-0b37eb06f2c3","Type":"ContainerDied","Data":"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"}
Sep 17 18:11:34.983039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.983016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp" event={"ID":"708095bb-43a1-4db6-afd2-0b37eb06f2c3","Type":"ContainerDied","Data":"67ef45208ee0bf3262602066ee2d326d8d89ab25a3307cf2b0f752d9816960d3"}
Sep 17 18:11:34.983039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.983039    2577 scope.go:117] "RemoveContainer" containerID="4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"
Sep 17 18:11:34.983250 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.983089    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"
Sep 17 18:11:34.991253 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.991233    2577 scope.go:117] "RemoveContainer" containerID="4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"
Sep 17 18:11:34.991512 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:11:34.991495    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f\": container with ID starting with 4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f not found: ID does not exist" containerID="4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"
Sep 17 18:11:34.991613 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:34.991518    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f"} err="failed to get container status \"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f\": rpc error: code = NotFound desc = could not find container \"4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f\": container with ID starting with 4e310fa5904db8b95d57b26adc21fc3a3fa874c41feca446ea319808042ee21f not found: ID does not exist"
Sep 17 18:11:35.002827 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.002805    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:11:35.007096 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.007076    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dv2j9p"]
Sep 17 18:11:35.029751 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.029727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb5n9\" (UniqueName: \"kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9\") pod \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") "
Sep 17 18:11:35.029831 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.029789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs\") pod \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\" (UID: \"708095bb-43a1-4db6-afd2-0b37eb06f2c3\") "
Sep 17 18:11:35.031801 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.031776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "708095bb-43a1-4db6-afd2-0b37eb06f2c3" (UID: "708095bb-43a1-4db6-afd2-0b37eb06f2c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:35.031877 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.031813    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9" (OuterVolumeSpecName: "kube-api-access-xb5n9") pod "708095bb-43a1-4db6-afd2-0b37eb06f2c3" (UID: "708095bb-43a1-4db6-afd2-0b37eb06f2c3"). InnerVolumeSpecName "kube-api-access-xb5n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:35.131053 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.131032    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb5n9\" (UniqueName: \"kubernetes.io/projected/708095bb-43a1-4db6-afd2-0b37eb06f2c3-kube-api-access-xb5n9\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.131053 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.131051    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708095bb-43a1-4db6-afd2-0b37eb06f2c3-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.307225 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.307201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:11:35.310896 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:35.310872    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schebrdwp"]
Sep 17 18:11:36.850797 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:36.850769    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" path="/var/lib/kubelet/pods/33c5a4d1-0c39-4b89-bb27-4813375f6b5b/volumes"
Sep 17 18:11:36.851194 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:36.851182    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" path="/var/lib/kubelet/pods/708095bb-43a1-4db6-afd2-0b37eb06f2c3/volumes"
Sep 17 18:11:39.843040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:39.843005    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:11:45.338316 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:45.338273    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:11:54.864053 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:54.864009    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main" containerID="cri-o://026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763" gracePeriod=45
Sep 17 18:11:55.338677 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:11:55.338588    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:05.338791 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:05.338745    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:05.339238 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:05.338883    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:12:15.339286 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:15.339239    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:25.116123 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.116088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:12:25.220779 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.220683    2577 generic.go:356] "Generic (PLEG): container finished" podID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerID="026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763" exitCode=0
Sep 17 18:12:25.220779 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.220752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"
Sep 17 18:12:25.220990 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.220771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" event={"ID":"6888c6c4-2f19-456d-b9fc-349abb6a385d","Type":"ContainerDied","Data":"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"}
Sep 17 18:12:25.220990 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.220863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6" event={"ID":"6888c6c4-2f19-456d-b9fc-349abb6a385d","Type":"ContainerDied","Data":"09f3f04efd77a97e652517d4e29997275e3cac7986af5c4a9801ccd2d009bb7d"}
Sep 17 18:12:25.220990 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.220893    2577 scope.go:117] "RemoveContainer" containerID="026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"
Sep 17 18:12:25.229551 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.229521    2577 scope.go:117] "RemoveContainer" containerID="026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"
Sep 17 18:12:25.229900 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:12:25.229869    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763\": container with ID starting with 026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763 not found: ID does not exist" containerID="026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"
Sep 17 18:12:25.229989 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.229908    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763"} err="failed to get container status \"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763\": rpc error: code = NotFound desc = could not find container \"026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763\": container with ID starting with 026a85a82b769011eea64869ddb866bccbd7e0c4f849ad6647f7b4d2848f8763 not found: ID does not exist"
Sep 17 18:12:25.231111 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.231091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs\") pod \"6888c6c4-2f19-456d-b9fc-349abb6a385d\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") "
Sep 17 18:12:25.231163 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.231153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6gd\" (UniqueName: \"kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd\") pod \"6888c6c4-2f19-456d-b9fc-349abb6a385d\" (UID: \"6888c6c4-2f19-456d-b9fc-349abb6a385d\") "
Sep 17 18:12:25.233350 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.233319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd" (OuterVolumeSpecName: "kube-api-access-nh6gd") pod "6888c6c4-2f19-456d-b9fc-349abb6a385d" (UID: "6888c6c4-2f19-456d-b9fc-349abb6a385d"). InnerVolumeSpecName "kube-api-access-nh6gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:25.233350 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.233333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6888c6c4-2f19-456d-b9fc-349abb6a385d" (UID: "6888c6c4-2f19-456d-b9fc-349abb6a385d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:25.332081 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.332038    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6888c6c4-2f19-456d-b9fc-349abb6a385d-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:12:25.332081 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.332072    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6gd\" (UniqueName: \"kubernetes.io/projected/6888c6c4-2f19-456d-b9fc-349abb6a385d-kube-api-access-nh6gd\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:12:25.544167 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.544125    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:12:25.547999 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:25.547961    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7hpz6"]
Sep 17 18:12:26.850382 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:26.850339    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" path="/var/lib/kubelet/pods/6888c6c4-2f19-456d-b9fc-349abb6a385d/volumes"
Sep 17 18:12:41.242013 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.241921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242282    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar-init"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar-init"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242306    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242319    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242325    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242331    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242336    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242355    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242418    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6888c6c4-2f19-456d-b9fc-349abb6a385d" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242428    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="708095bb-43a1-4db6-afd2-0b37eb06f2c3" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="main"
Sep 17 18:12:41.242440 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.242442    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c5a4d1-0c39-4b89-bb27-4813375f6b5b" containerName="modelcar"
Sep 17 18:12:41.247004 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.246980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.250362 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.250328    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 18:12:41.250564 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.250417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-gpdpt\""
Sep 17 18:12:41.250564 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.250496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 18:12:41.250564 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.250502    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-prhbl\""
Sep 17 18:12:41.250739 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.250623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 18:12:41.259583 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.259552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:12:41.361733 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.361686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.361927 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.361751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5fh\" (UniqueName: \"kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.462698 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.462650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.462893 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.462709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5fh\" (UniqueName: \"kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.465366 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.465342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.477074 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.477037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5fh\" (UniqueName: \"kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.559846 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.559738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:41.841255 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:41.841162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:12:41.844745 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:12:41.844711    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fe35b8_574c_4348_83b6_50ee68b37ab0.slice/crio-c764eb04c5bf7cf22a500702bef8d6cea958a1b60570a2aab094e6acb9a70c8b WatchSource:0}: Error finding container c764eb04c5bf7cf22a500702bef8d6cea958a1b60570a2aab094e6acb9a70c8b: Status 404 returned error can't find the container with id c764eb04c5bf7cf22a500702bef8d6cea958a1b60570a2aab094e6acb9a70c8b
Sep 17 18:12:42.304839 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:42.304800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" event={"ID":"22fe35b8-574c-4348-83b6-50ee68b37ab0","Type":"ContainerStarted","Data":"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"}
Sep 17 18:12:42.305357 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:42.304859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" event={"ID":"22fe35b8-574c-4348-83b6-50ee68b37ab0","Type":"ContainerStarted","Data":"c764eb04c5bf7cf22a500702bef8d6cea958a1b60570a2aab094e6acb9a70c8b"}
Sep 17 18:12:42.305357 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:42.304888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:12:42.320854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:12:42.320801    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podStartSLOduration=1.320780264 podStartE2EDuration="1.320780264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:42.320369256 +0000 UTC m=+2337.978844257" watchObservedRunningTime="2026-09-17 18:12:42.320780264 +0000 UTC m=+2337.979255265"
Sep 17 18:13:13.315156 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:13:13.315117    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:13:45.584376 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:13:45.584338    2577 scope.go:117] "RemoveContainer" containerID="5e177744e5e0f2dbc58d4f06f58b3a7e5849419a9c7ea7e30d6327bf49ffd68c"
Sep 17 18:13:45.596323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:13:45.596293    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:13:45.598874 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:13:45.598840    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:14:26.562196 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:26.562145    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:14:33.315485 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:33.315441    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:41.586105 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:41.586063    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main" containerID="cri-o://b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7" gracePeriod=45
Sep 17 18:14:43.314904 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:43.314861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:45.607651 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:45.607610    2577 scope.go:117] "RemoveContainer" containerID="16adb99f30c96d576e5fb63a3d41b5b8d2f350d0ca7484788b6ee522508c8c22"
Sep 17 18:14:53.314991 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:53.314952    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:53.315426 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:14:53.315061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:15:03.315008 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:03.314962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:11.846683 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.846651    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:15:11.898682 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.898640    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs\") pod \"22fe35b8-574c-4348-83b6-50ee68b37ab0\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") "
Sep 17 18:15:11.898891 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.898709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc5fh\" (UniqueName: \"kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh\") pod \"22fe35b8-574c-4348-83b6-50ee68b37ab0\" (UID: \"22fe35b8-574c-4348-83b6-50ee68b37ab0\") "
Sep 17 18:15:11.901114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.901063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22fe35b8-574c-4348-83b6-50ee68b37ab0" (UID: "22fe35b8-574c-4348-83b6-50ee68b37ab0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:11.901114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.901096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh" (OuterVolumeSpecName: "kube-api-access-xc5fh") pod "22fe35b8-574c-4348-83b6-50ee68b37ab0" (UID: "22fe35b8-574c-4348-83b6-50ee68b37ab0"). InnerVolumeSpecName "kube-api-access-xc5fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:11.999379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.999332    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5fh\" (UniqueName: \"kubernetes.io/projected/22fe35b8-574c-4348-83b6-50ee68b37ab0-kube-api-access-xc5fh\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:15:11.999379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:11.999370    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22fe35b8-574c-4348-83b6-50ee68b37ab0-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:15:12.055594 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.055553    2577 generic.go:356] "Generic (PLEG): container finished" podID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerID="b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7" exitCode=0
Sep 17 18:15:12.055788 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.055602    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"
Sep 17 18:15:12.055788 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.055634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" event={"ID":"22fe35b8-574c-4348-83b6-50ee68b37ab0","Type":"ContainerDied","Data":"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"}
Sep 17 18:15:12.055788 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.055708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c" event={"ID":"22fe35b8-574c-4348-83b6-50ee68b37ab0","Type":"ContainerDied","Data":"c764eb04c5bf7cf22a500702bef8d6cea958a1b60570a2aab094e6acb9a70c8b"}
Sep 17 18:15:12.055788 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.055731    2577 scope.go:117] "RemoveContainer" containerID="b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"
Sep 17 18:15:12.064502 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.064479    2577 scope.go:117] "RemoveContainer" containerID="b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"
Sep 17 18:15:12.064849 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:15:12.064829    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7\": container with ID starting with b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7 not found: ID does not exist" containerID="b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"
Sep 17 18:15:12.064918 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.064858    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7"} err="failed to get container status \"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7\": rpc error: code = NotFound desc = could not find container \"b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7\": container with ID starting with b1c590e4c83b224f8f85634ab7901498dca3e70b8c2845d7b64d000d363e75d7 not found: ID does not exist"
Sep 17 18:15:12.076960 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.076917    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:15:12.081288 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.081250    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schevpb7c"]
Sep 17 18:15:12.850771 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:12.850730    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" path="/var/lib/kubelet/pods/22fe35b8-574c-4348-83b6-50ee68b37ab0/volumes"
Sep 17 18:15:38.084452 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.084336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:15:38.084936 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.084702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main"
Sep 17 18:15:38.084936 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.084717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main"
Sep 17 18:15:38.084936 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.084788    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fe35b8-574c-4348-83b6-50ee68b37ab0" containerName="main"
Sep 17 18:15:38.087752 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.087724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.090799 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.090770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:15:38.090993 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.090771    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-ts8kf\""
Sep 17 18:15:38.090993 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.090814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:15:38.090993 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.090825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-mdnqd\""
Sep 17 18:15:38.090993 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.090770    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:15:38.096640 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.096610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:15:38.205878 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.205836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.206109 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.205977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dq7z\" (UniqueName: \"kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.307256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.307210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dq7z\" (UniqueName: \"kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.307256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.307259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.309838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.309814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.317511 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.317471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dq7z\" (UniqueName: \"kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.400478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.400431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:38.676969 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:38.676914    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:15:38.680000 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:15:38.679967    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603d9426_379f_41b5_878c_d86b2f47dab0.slice/crio-e47c89df78f823fa06a79737478e99b396ea82c486fcf9088e6650ab3dc835fc WatchSource:0}: Error finding container e47c89df78f823fa06a79737478e99b396ea82c486fcf9088e6650ab3dc835fc: Status 404 returned error can't find the container with id e47c89df78f823fa06a79737478e99b396ea82c486fcf9088e6650ab3dc835fc
Sep 17 18:15:39.189042 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:39.189002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" event={"ID":"603d9426-379f-41b5-878c-d86b2f47dab0","Type":"ContainerStarted","Data":"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"}
Sep 17 18:15:39.189521 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:39.189061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" event={"ID":"603d9426-379f-41b5-878c-d86b2f47dab0","Type":"ContainerStarted","Data":"e47c89df78f823fa06a79737478e99b396ea82c486fcf9088e6650ab3dc835fc"}
Sep 17 18:15:39.189521 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:39.189135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:15:39.206736 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:39.206644    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podStartSLOduration=1.20661337 podStartE2EDuration="1.20661337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:39.205537529 +0000 UTC m=+2514.864012530" watchObservedRunningTime="2026-09-17 18:15:39.20661337 +0000 UTC m=+2514.865088373"
Sep 17 18:15:45.629563 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:45.629516    2577 scope.go:117] "RemoveContainer" containerID="d6909b5b00897bedfb0b9746ffdc69a5196af1857d58e2e7f5ce370d4759e81f"
Sep 17 18:15:45.638518 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:45.638487    2577 scope.go:117] "RemoveContainer" containerID="b0b978d91b7ba4b5687c2b2828b8b30bf9544d2048cbe2d0725190ea7b13f4d7"
Sep 17 18:15:58.122247 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.122206    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:15:58.125839 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.125808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.129017 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.128981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-dthlj\""
Sep 17 18:15:58.129017 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.129005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-2w49v\""
Sep 17 18:15:58.129265 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.128992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:15:58.129265 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.128992    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:15:58.129265 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.129040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:15:58.133281 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.133249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:15:58.276439 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.276365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.276677 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.276488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxjg\" (UniqueName: \"kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.377835 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.377722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.377835 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.377799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxjg\" (UniqueName: \"kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.380689 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.380650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.386164 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.386131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxjg\" (UniqueName: \"kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.439455 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.439379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:58.717507 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:58.717477    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:15:58.720312 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:15:58.720277    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0c2630_5665_45f8_bd3e_0dceaa683b2d.slice/crio-194fc8a9feb07a63646549093aa79a2b46d8726ced38974047c13bae5d2dab72 WatchSource:0}: Error finding container 194fc8a9feb07a63646549093aa79a2b46d8726ced38974047c13bae5d2dab72: Status 404 returned error can't find the container with id 194fc8a9feb07a63646549093aa79a2b46d8726ced38974047c13bae5d2dab72
Sep 17 18:15:59.285668 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:59.285627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" event={"ID":"bb0c2630-5665-45f8-bd3e-0dceaa683b2d","Type":"ContainerStarted","Data":"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"}
Sep 17 18:15:59.286080 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:59.285690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" event={"ID":"bb0c2630-5665-45f8-bd3e-0dceaa683b2d","Type":"ContainerStarted","Data":"194fc8a9feb07a63646549093aa79a2b46d8726ced38974047c13bae5d2dab72"}
Sep 17 18:15:59.286080 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:59.285715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:15:59.300992 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:15:59.300920    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podStartSLOduration=1.300898882 podStartE2EDuration="1.300898882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:59.300191865 +0000 UTC m=+2534.958666864" watchObservedRunningTime="2026-09-17 18:15:59.300898882 +0000 UTC m=+2534.959373883"
Sep 17 18:16:10.196707 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:16:10.196674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:16:30.292622 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:16:30.292590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:17:20.213325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:20.213272    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:17:20.293080 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:20.293039    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:30.292270 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:30.292223    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:35.234049 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:35.233999    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" containerID="cri-o://6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc" gracePeriod=45
Sep 17 18:17:40.292425 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:40.292368    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:40.292858 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:40.292492    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:17:42.989073 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:42.989020    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:17:50.196002 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:50.195958    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:50.292503 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:50.292458    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:58.009631 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:17:58.009573    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main" containerID="cri-o://35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3" gracePeriod=45
Sep 17 18:18:00.196361 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:00.196316    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:00.292751 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:00.292709    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:05.492744 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.492708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:18:05.537270 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.537173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs\") pod \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") "
Sep 17 18:18:05.537270 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.537225    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcxjg\" (UniqueName: \"kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg\") pod \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\" (UID: \"bb0c2630-5665-45f8-bd3e-0dceaa683b2d\") "
Sep 17 18:18:05.539649 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.539615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb0c2630-5665-45f8-bd3e-0dceaa683b2d" (UID: "bb0c2630-5665-45f8-bd3e-0dceaa683b2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:05.539649 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.539625    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg" (OuterVolumeSpecName: "kube-api-access-lcxjg") pod "bb0c2630-5665-45f8-bd3e-0dceaa683b2d" (UID: "bb0c2630-5665-45f8-bd3e-0dceaa683b2d"). InnerVolumeSpecName "kube-api-access-lcxjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:05.637988 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.637938    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxjg\" (UniqueName: \"kubernetes.io/projected/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-kube-api-access-lcxjg\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:18:05.637988 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.637974    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0c2630-5665-45f8-bd3e-0dceaa683b2d-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:18:05.901366 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.901324    2577 generic.go:356] "Generic (PLEG): container finished" podID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerID="6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc" exitCode=0
Sep 17 18:18:05.901563 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.901375    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"
Sep 17 18:18:05.901563 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.901416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" event={"ID":"bb0c2630-5665-45f8-bd3e-0dceaa683b2d","Type":"ContainerDied","Data":"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"}
Sep 17 18:18:05.901563 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.901466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj" event={"ID":"bb0c2630-5665-45f8-bd3e-0dceaa683b2d","Type":"ContainerDied","Data":"194fc8a9feb07a63646549093aa79a2b46d8726ced38974047c13bae5d2dab72"}
Sep 17 18:18:05.901563 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.901489    2577 scope.go:117] "RemoveContainer" containerID="6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"
Sep 17 18:18:05.911037 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.911015    2577 scope.go:117] "RemoveContainer" containerID="6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"
Sep 17 18:18:05.911425 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:18:05.911380    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc\": container with ID starting with 6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc not found: ID does not exist" containerID="6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"
Sep 17 18:18:05.911535 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.911445    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc"} err="failed to get container status \"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc\": rpc error: code = NotFound desc = could not find container \"6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc\": container with ID starting with 6e7a9cb14d41f7500035970774fc01aa0bbce1358fdec93400f08e29dc002ecc not found: ID does not exist"
Sep 17 18:18:05.924109 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.924073    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:18:05.929976 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:05.929941    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schebmjhj"]
Sep 17 18:18:06.851323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:06.851285    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" path="/var/lib/kubelet/pods/bb0c2630-5665-45f8-bd3e-0dceaa683b2d/volumes"
Sep 17 18:18:10.196674 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:10.196628    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:10.197129 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:10.196752    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:18:20.196424 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:20.196355    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:26.034474 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.034432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:18:26.035059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.034949    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main"
Sep 17 18:18:26.035059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.034969    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main"
Sep 17 18:18:26.035183 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.035075    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0c2630-5665-45f8-bd3e-0dceaa683b2d" containerName="main"
Sep 17 18:18:26.038046 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.038021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.040986 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.040956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-ptfh6\""
Sep 17 18:18:26.041178 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.040988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:18:26.041266 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.041028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:18:26.041266 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.041198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:18:26.041266 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.041115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-gpz8m\""
Sep 17 18:18:26.047157 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.047130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:18:26.108974 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.108936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msv5\" (UniqueName: \"kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.108974 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.108989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.210423 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.210357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.210633 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.210485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msv5\" (UniqueName: \"kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.213174 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.213139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.219543 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.219513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msv5\" (UniqueName: \"kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.350303 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.350186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:26.627165 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:26.627137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:18:26.629935 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:18:26.629901    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b79256_03b1_4a06_bbb1_d1b3cc242600.slice/crio-3407fd43863889162f17033908faa24614384518dd3b0b921cdc1aa969996695 WatchSource:0}: Error finding container 3407fd43863889162f17033908faa24614384518dd3b0b921cdc1aa969996695: Status 404 returned error can't find the container with id 3407fd43863889162f17033908faa24614384518dd3b0b921cdc1aa969996695
Sep 17 18:18:27.008908 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:27.008800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" event={"ID":"72b79256-03b1-4a06-bbb1-d1b3cc242600","Type":"ContainerStarted","Data":"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"}
Sep 17 18:18:27.008908 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:27.008859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" event={"ID":"72b79256-03b1-4a06-bbb1-d1b3cc242600","Type":"ContainerStarted","Data":"3407fd43863889162f17033908faa24614384518dd3b0b921cdc1aa969996695"}
Sep 17 18:18:27.008908 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:27.008892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:18:27.026695 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:27.026616    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podStartSLOduration=1.026595929 podStartE2EDuration="1.026595929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:27.025681953 +0000 UTC m=+2682.684156957" watchObservedRunningTime="2026-09-17 18:18:27.026595929 +0000 UTC m=+2682.685070926"
Sep 17 18:18:28.263198 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.263166    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:18:28.327982 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.327879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dq7z\" (UniqueName: \"kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z\") pod \"603d9426-379f-41b5-878c-d86b2f47dab0\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") "
Sep 17 18:18:28.327982 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.327943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs\") pod \"603d9426-379f-41b5-878c-d86b2f47dab0\" (UID: \"603d9426-379f-41b5-878c-d86b2f47dab0\") "
Sep 17 18:18:28.330312 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.330278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "603d9426-379f-41b5-878c-d86b2f47dab0" (UID: "603d9426-379f-41b5-878c-d86b2f47dab0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:28.330471 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.330279    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z" (OuterVolumeSpecName: "kube-api-access-6dq7z") pod "603d9426-379f-41b5-878c-d86b2f47dab0" (UID: "603d9426-379f-41b5-878c-d86b2f47dab0"). InnerVolumeSpecName "kube-api-access-6dq7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:28.429451 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.429376    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dq7z\" (UniqueName: \"kubernetes.io/projected/603d9426-379f-41b5-878c-d86b2f47dab0-kube-api-access-6dq7z\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:18:28.429451 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:28.429443    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/603d9426-379f-41b5-878c-d86b2f47dab0-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:18:29.020464 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.020424    2577 generic.go:356] "Generic (PLEG): container finished" podID="603d9426-379f-41b5-878c-d86b2f47dab0" containerID="35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3" exitCode=0
Sep 17 18:18:29.020464 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.020468    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"
Sep 17 18:18:29.020727 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.020474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" event={"ID":"603d9426-379f-41b5-878c-d86b2f47dab0","Type":"ContainerDied","Data":"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"}
Sep 17 18:18:29.020727 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.020518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b" event={"ID":"603d9426-379f-41b5-878c-d86b2f47dab0","Type":"ContainerDied","Data":"e47c89df78f823fa06a79737478e99b396ea82c486fcf9088e6650ab3dc835fc"}
Sep 17 18:18:29.020727 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.020544    2577 scope.go:117] "RemoveContainer" containerID="35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"
Sep 17 18:18:29.029438 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.029375    2577 scope.go:117] "RemoveContainer" containerID="35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"
Sep 17 18:18:29.029812 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:18:29.029787    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3\": container with ID starting with 35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3 not found: ID does not exist" containerID="35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"
Sep 17 18:18:29.029905 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.029827    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3"} err="failed to get container status \"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3\": rpc error: code = NotFound desc = could not find container \"35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3\": container with ID starting with 35ed27cfdf32486ddf4ed6fd5d1be906cf891b5beda66abeb0905a942e3ea6e3 not found: ID does not exist"
Sep 17 18:18:29.036963 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.036927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:18:29.041447 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:29.041415    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-s5b7b"]
Sep 17 18:18:30.850527 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:30.850490    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" path="/var/lib/kubelet/pods/603d9426-379f-41b5-878c-d86b2f47dab0/volumes"
Sep 17 18:18:45.622658 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:45.622624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:18:45.627192 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:45.627156    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:18:58.016521 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:18:58.016484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:20:27.893412 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:27.893358    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:20:28.017146 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:28.017097    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:38.016379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:38.016336    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:42.916049 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:42.915986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" containerID="cri-o://27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61" gracePeriod=45
Sep 17 18:20:48.016238 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:48.016196    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:48.016682 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:48.016306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:20:54.116533 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.116492    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:20:54.116942 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.116854    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main"
Sep 17 18:20:54.116942 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.116867    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main"
Sep 17 18:20:54.116942 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.116934    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="603d9426-379f-41b5-878c-d86b2f47dab0" containerName="main"
Sep 17 18:20:54.120276 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.120248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.122494 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.122466    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-8jg7s\""
Sep 17 18:20:54.128523 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.128484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:20:54.233180 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.233134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvs77\" (UniqueName: \"kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.233383 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.233204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.333719 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.333672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvs77\" (UniqueName: \"kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.333930 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.333737    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.336408 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.336355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.342663 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.342628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvs77\" (UniqueName: \"kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.433414 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.433284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:54.710291 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.710260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:20:54.713111 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:20:54.713072    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadf35df_7d5e_4b5d_8862_9935bfaea4a0.slice/crio-fd766f2a889df727f23ed79e7e4f305803a3e74e43655dc16435ff11ebef6142 WatchSource:0}: Error finding container fd766f2a889df727f23ed79e7e4f305803a3e74e43655dc16435ff11ebef6142: Status 404 returned error can't find the container with id fd766f2a889df727f23ed79e7e4f305803a3e74e43655dc16435ff11ebef6142
Sep 17 18:20:54.739603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:54.739564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" event={"ID":"dadf35df-7d5e-4b5d-8862-9935bfaea4a0","Type":"ContainerStarted","Data":"fd766f2a889df727f23ed79e7e4f305803a3e74e43655dc16435ff11ebef6142"}
Sep 17 18:20:55.746276 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:55.746230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" event={"ID":"dadf35df-7d5e-4b5d-8862-9935bfaea4a0","Type":"ContainerStarted","Data":"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"}
Sep 17 18:20:55.746744 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:55.746350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:20:55.762674 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:55.762606    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podStartSLOduration=1.762587344 podStartE2EDuration="1.762587344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:55.762278637 +0000 UTC m=+2831.420753662" watchObservedRunningTime="2026-09-17 18:20:55.762587344 +0000 UTC m=+2831.421062345"
Sep 17 18:20:58.016972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:20:58.016927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:08.016558 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:08.016514    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:13.175170 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.175078    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:21:13.300010 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.299971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5msv5\" (UniqueName: \"kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5\") pod \"72b79256-03b1-4a06-bbb1-d1b3cc242600\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") "
Sep 17 18:21:13.300199 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.300088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs\") pod \"72b79256-03b1-4a06-bbb1-d1b3cc242600\" (UID: \"72b79256-03b1-4a06-bbb1-d1b3cc242600\") "
Sep 17 18:21:13.302546 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.302505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72b79256-03b1-4a06-bbb1-d1b3cc242600" (UID: "72b79256-03b1-4a06-bbb1-d1b3cc242600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:13.302546 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.302510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5" (OuterVolumeSpecName: "kube-api-access-5msv5") pod "72b79256-03b1-4a06-bbb1-d1b3cc242600" (UID: "72b79256-03b1-4a06-bbb1-d1b3cc242600"). InnerVolumeSpecName "kube-api-access-5msv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:13.400911 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.400862    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72b79256-03b1-4a06-bbb1-d1b3cc242600-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:21:13.400911 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.400902    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5msv5\" (UniqueName: \"kubernetes.io/projected/72b79256-03b1-4a06-bbb1-d1b3cc242600-kube-api-access-5msv5\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:21:13.833458 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.833367    2577 generic.go:356] "Generic (PLEG): container finished" podID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerID="27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61" exitCode=0
Sep 17 18:21:13.833655 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.833456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" event={"ID":"72b79256-03b1-4a06-bbb1-d1b3cc242600","Type":"ContainerDied","Data":"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"}
Sep 17 18:21:13.833655 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.833436    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"
Sep 17 18:21:13.833655 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.833512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr" event={"ID":"72b79256-03b1-4a06-bbb1-d1b3cc242600","Type":"ContainerDied","Data":"3407fd43863889162f17033908faa24614384518dd3b0b921cdc1aa969996695"}
Sep 17 18:21:13.833655 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.833535    2577 scope.go:117] "RemoveContainer" containerID="27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"
Sep 17 18:21:13.842665 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.842642    2577 scope.go:117] "RemoveContainer" containerID="27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"
Sep 17 18:21:13.842997 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:21:13.842978    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61\": container with ID starting with 27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61 not found: ID does not exist" containerID="27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"
Sep 17 18:21:13.843063 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.843007    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61"} err="failed to get container status \"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61\": rpc error: code = NotFound desc = could not find container \"27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61\": container with ID starting with 27bb1c3cf1d26a00526f1ba3d005adf92380d5500fe5926ef1c1a3367a473a61 not found: ID does not exist"
Sep 17 18:21:13.857061 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.857023    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:21:13.861779 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:13.861743    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-4b2dr"]
Sep 17 18:21:14.851469 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:14.851428    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" path="/var/lib/kubelet/pods/72b79256-03b1-4a06-bbb1-d1b3cc242600/volumes"
Sep 17 18:21:26.753560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:21:26.753522    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:22:55.781678 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:22:55.781638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:22:56.753635 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:22:56.753586    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:06.754365 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:06.754324    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:10.805918 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:10.805824    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" containerID="cri-o://a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7" gracePeriod=45
Sep 17 18:23:16.753937 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:16.753891    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:16.754466 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:16.754011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:23:26.754039 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:26.753997    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:36.753519 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:36.753477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:41.064478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.064355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:23:41.185525 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.185476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs\") pod \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") "
Sep 17 18:23:41.185717 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.185559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvs77\" (UniqueName: \"kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77\") pod \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\" (UID: \"dadf35df-7d5e-4b5d-8862-9935bfaea4a0\") "
Sep 17 18:23:41.187842 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.187805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77" (OuterVolumeSpecName: "kube-api-access-gvs77") pod "dadf35df-7d5e-4b5d-8862-9935bfaea4a0" (UID: "dadf35df-7d5e-4b5d-8862-9935bfaea4a0"). InnerVolumeSpecName "kube-api-access-gvs77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:41.187842 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.187816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dadf35df-7d5e-4b5d-8862-9935bfaea4a0" (UID: "dadf35df-7d5e-4b5d-8862-9935bfaea4a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:41.287054 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.287007    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:23:41.287054 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.287047    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvs77\" (UniqueName: \"kubernetes.io/projected/dadf35df-7d5e-4b5d-8862-9935bfaea4a0-kube-api-access-gvs77\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:23:41.534268 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.534231    2577 generic.go:356] "Generic (PLEG): container finished" podID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerID="a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7" exitCode=0
Sep 17 18:23:41.534478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.534282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"
Sep 17 18:23:41.534478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.534278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" event={"ID":"dadf35df-7d5e-4b5d-8862-9935bfaea4a0","Type":"ContainerDied","Data":"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"}
Sep 17 18:23:41.534478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.534416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5" event={"ID":"dadf35df-7d5e-4b5d-8862-9935bfaea4a0","Type":"ContainerDied","Data":"fd766f2a889df727f23ed79e7e4f305803a3e74e43655dc16435ff11ebef6142"}
Sep 17 18:23:41.534478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.534447    2577 scope.go:117] "RemoveContainer" containerID="a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"
Sep 17 18:23:41.544308 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.544283    2577 scope.go:117] "RemoveContainer" containerID="a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"
Sep 17 18:23:41.544684 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:23:41.544663    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7\": container with ID starting with a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7 not found: ID does not exist" containerID="a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"
Sep 17 18:23:41.544746 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.544693    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7"} err="failed to get container status \"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7\": rpc error: code = NotFound desc = could not find container \"a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7\": container with ID starting with a7d363a1ccaba3946333a66f02a60e82c2b7c704da41d065124ce022af9b92e7 not found: ID does not exist"
Sep 17 18:23:41.556339 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.556297    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:23:41.558574 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:41.558541    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-7b2q5"]
Sep 17 18:23:42.850441 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:42.850378    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" path="/var/lib/kubelet/pods/dadf35df-7d5e-4b5d-8862-9935bfaea4a0/volumes"
Sep 17 18:23:45.649372 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:45.649236    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:23:45.656972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:23:45.656935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:26:03.314884 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.314846    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315193    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main"
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315205    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main"
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315223    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main"
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main"
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315287    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b79256-03b1-4a06-bbb1-d1b3cc242600" containerName="main"
Sep 17 18:26:03.315370 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.315295    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadf35df-7d5e-4b5d-8862-9935bfaea4a0" containerName="main"
Sep 17 18:26:03.318347 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.318326    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.321615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.321580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:26:03.322341 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.322321    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-tqvwh\""
Sep 17 18:26:03.322795 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.322778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:26:03.326150 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.326130    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:26:03.340543 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.340505    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:26:03.426954 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.426916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.426954 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.426957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.426983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqsd\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427242 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.427626 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.427252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.527826 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqsd\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.527968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528365 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528365 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528365 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528567 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528649 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528707 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.528781 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.528763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.530438 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.530409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.530863 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.530841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.536529 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.536502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.536777 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.536754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqsd\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd\") pod \"router-gateway-2-openshift-default-78888675c6-cb9bv\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.629850 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.629800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:03.911225 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.911080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:26:03.914202 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:26:03.914163    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a75af94_3ad6_4701_9ca0_f02b5b826301.slice/crio-c703cda6d89024e4e556e93c41fcd14d9dae52cbda5065e12750d784fc8e7bb6 WatchSource:0}: Error finding container c703cda6d89024e4e556e93c41fcd14d9dae52cbda5065e12750d784fc8e7bb6: Status 404 returned error can't find the container with id c703cda6d89024e4e556e93c41fcd14d9dae52cbda5065e12750d784fc8e7bb6
Sep 17 18:26:03.916470 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.916422    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:26:03.916583 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.916521    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:26:03.916583 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:03.916562    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 18:26:04.202443 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:04.202294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" event={"ID":"9a75af94-3ad6-4701-9ca0-f02b5b826301","Type":"ContainerStarted","Data":"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"}
Sep 17 18:26:04.202443 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:04.202358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" event={"ID":"9a75af94-3ad6-4701-9ca0-f02b5b826301","Type":"ContainerStarted","Data":"c703cda6d89024e4e556e93c41fcd14d9dae52cbda5065e12750d784fc8e7bb6"}
Sep 17 18:26:04.220974 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:04.220907    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podStartSLOduration=1.220884237 podStartE2EDuration="1.220884237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:04.220206518 +0000 UTC m=+3139.878681520" watchObservedRunningTime="2026-09-17 18:26:04.220884237 +0000 UTC m=+3139.879359242"
Sep 17 18:26:04.630719 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:04.630668    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:04.632339 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:04.632303    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:05.630217 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:05.630161    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:06.630863 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:06.630813    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:07.630596 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:07.630502    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:08.630421 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:08.630346    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:09.630910 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:09.630866    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 17 18:26:10.635050 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:10.635010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:10.635517 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:10.635302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:10.636112 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:10.636089    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:26:18.417620 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.417573    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:26:18.421243 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.421213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.424379 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.424342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:26:18.424584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.424417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-zbb9p\""
Sep 17 18:26:18.424584 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.424436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-cb7lh\""
Sep 17 18:26:18.431850 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.431812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:26:18.565436 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.565367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.565436 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.565441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75jh\" (UniqueName: \"kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.665974 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.665930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.666167 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.665983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75jh\" (UniqueName: \"kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.668864 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.668789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.674873 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.674827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75jh\" (UniqueName: \"kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:18.733663 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:18.733620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:19.016289 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:19.016257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:26:19.018842 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:26:19.018810    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aaab282_5173_41bd_86ab_fb77c4d1556c.slice/crio-53175ec2d863994dba58131fe8c6c4efcecb77a5846917feac1afdb3de6c9c3c WatchSource:0}: Error finding container 53175ec2d863994dba58131fe8c6c4efcecb77a5846917feac1afdb3de6c9c3c: Status 404 returned error can't find the container with id 53175ec2d863994dba58131fe8c6c4efcecb77a5846917feac1afdb3de6c9c3c
Sep 17 18:26:19.270916 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:19.270819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" event={"ID":"4aaab282-5173-41bd-86ab-fb77c4d1556c","Type":"ContainerStarted","Data":"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"}
Sep 17 18:26:19.270916 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:19.270874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" event={"ID":"4aaab282-5173-41bd-86ab-fb77c4d1556c","Type":"ContainerStarted","Data":"53175ec2d863994dba58131fe8c6c4efcecb77a5846917feac1afdb3de6c9c3c"}
Sep 17 18:26:19.271114 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:19.270928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:26:19.287620 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:19.287560    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podStartSLOduration=1.287540962 podStartE2EDuration="1.287540962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:19.286575236 +0000 UTC m=+3154.945050237" watchObservedRunningTime="2026-09-17 18:26:19.287540962 +0000 UTC m=+3154.946015962"
Sep 17 18:26:50.277940 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:26:50.277908    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:28:28.162123 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:28.162077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:28:30.277160 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:30.277117    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:28:40.277967 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:40.277925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:28:43.183540 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:43.183474    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" containerID="cri-o://0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802" gracePeriod=45
Sep 17 18:28:45.679425 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:45.679283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:28:45.684547 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:45.683451    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:28:50.277127 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:50.277084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:28:50.277573 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:28:50.277228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:29:00.277787 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:00.277742    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:10.277995 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:10.277904    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:13.435078 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.435044    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:29:13.523741 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.523634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l75jh\" (UniqueName: \"kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh\") pod \"4aaab282-5173-41bd-86ab-fb77c4d1556c\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") "
Sep 17 18:29:13.523741 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.523722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs\") pod \"4aaab282-5173-41bd-86ab-fb77c4d1556c\" (UID: \"4aaab282-5173-41bd-86ab-fb77c4d1556c\") "
Sep 17 18:29:13.526059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.526026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4aaab282-5173-41bd-86ab-fb77c4d1556c" (UID: "4aaab282-5173-41bd-86ab-fb77c4d1556c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:13.526193 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.526033    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh" (OuterVolumeSpecName: "kube-api-access-l75jh") pod "4aaab282-5173-41bd-86ab-fb77c4d1556c" (UID: "4aaab282-5173-41bd-86ab-fb77c4d1556c"). InnerVolumeSpecName "kube-api-access-l75jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:13.625031 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.624974    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aaab282-5173-41bd-86ab-fb77c4d1556c-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:13.625031 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:13.625013    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l75jh\" (UniqueName: \"kubernetes.io/projected/4aaab282-5173-41bd-86ab-fb77c4d1556c-kube-api-access-l75jh\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:14.102626 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.102586    2577 generic.go:356] "Generic (PLEG): container finished" podID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerID="0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802" exitCode=0
Sep 17 18:29:14.102812 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.102646    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"
Sep 17 18:29:14.102812 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.102662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" event={"ID":"4aaab282-5173-41bd-86ab-fb77c4d1556c","Type":"ContainerDied","Data":"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"}
Sep 17 18:29:14.102812 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.102707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz" event={"ID":"4aaab282-5173-41bd-86ab-fb77c4d1556c","Type":"ContainerDied","Data":"53175ec2d863994dba58131fe8c6c4efcecb77a5846917feac1afdb3de6c9c3c"}
Sep 17 18:29:14.102812 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.102729    2577 scope.go:117] "RemoveContainer" containerID="0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"
Sep 17 18:29:14.111983 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.111964    2577 scope.go:117] "RemoveContainer" containerID="0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"
Sep 17 18:29:14.112306 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:29:14.112285    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802\": container with ID starting with 0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802 not found: ID does not exist" containerID="0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"
Sep 17 18:29:14.112360 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.112317    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802"} err="failed to get container status \"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802\": rpc error: code = NotFound desc = could not find container \"0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802\": container with ID starting with 0b3925203498660ee542c98bbf5ab54dcb3799b3de99bc74ea09229dcc6b4802 not found: ID does not exist"
Sep 17 18:29:14.124764 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.124724    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:29:14.126574 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.126545    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45sfqqz"]
Sep 17 18:29:14.852166 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:14.852124    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" path="/var/lib/kubelet/pods/4aaab282-5173-41bd-86ab-fb77c4d1556c/volumes"
Sep 17 18:29:23.242022 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:23.241980    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:29:23.242495 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:23.242205    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" containerID="cri-o://f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f" gracePeriod=30
Sep 17 18:29:25.637661 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:25.637619    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:29:28.499240 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.499208    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:29:28.654603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654603 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654678    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nqsd\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.654854 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo\") pod \"9a75af94-3ad6-4701-9ca0-f02b5b826301\" (UID: \"9a75af94-3ad6-4701-9ca0-f02b5b826301\") "
Sep 17 18:29:28.655296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:28.655296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.655124    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-credential-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.655296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.655110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:28.655296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.654941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:28.655538 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.655293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:28.655538 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.655130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:28.657082 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.657055    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:28.657082 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.657068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:28.657483 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.657462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:29:28.657557 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.657479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd" (OuterVolumeSpecName: "kube-api-access-7nqsd") pod "9a75af94-3ad6-4701-9ca0-f02b5b826301" (UID: "9a75af94-3ad6-4701-9ca0-f02b5b826301"). InnerVolumeSpecName "kube-api-access-7nqsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:28.756186 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756142    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nqsd\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-kube-api-access-7nqsd\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756186 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756179    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-data\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756186 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756189    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a75af94-3ad6-4701-9ca0-f02b5b826301-istiod-ca-cert\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756186 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756196    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-podinfo\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756186 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756204    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756214    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-token\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756222    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-istio-envoy\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:28.756560 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:28.756230    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a75af94-3ad6-4701-9ca0-f02b5b826301-workload-socket\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:29:29.174741 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.174633    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerID="f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f" exitCode=0
Sep 17 18:29:29.174741 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.174682    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"
Sep 17 18:29:29.174741 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.174721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" event={"ID":"9a75af94-3ad6-4701-9ca0-f02b5b826301","Type":"ContainerDied","Data":"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"}
Sep 17 18:29:29.175040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.174785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv" event={"ID":"9a75af94-3ad6-4701-9ca0-f02b5b826301","Type":"ContainerDied","Data":"c703cda6d89024e4e556e93c41fcd14d9dae52cbda5065e12750d784fc8e7bb6"}
Sep 17 18:29:29.175040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.174813    2577 scope.go:117] "RemoveContainer" containerID="f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"
Sep 17 18:29:29.185527 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.185470    2577 scope.go:117] "RemoveContainer" containerID="f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"
Sep 17 18:29:29.185965 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:29:29.185945    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f\": container with ID starting with f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f not found: ID does not exist" containerID="f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"
Sep 17 18:29:29.186054 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.185973    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f"} err="failed to get container status \"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f\": rpc error: code = NotFound desc = could not find container \"f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f\": container with ID starting with f7999f3f9b44d3aee7bbb9b249f0b1259fe4e59c50249d73d91ea7a24c25201f not found: ID does not exist"
Sep 17 18:29:29.192712 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.192673    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:29:29.198323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:29.198280    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-78888675c6-cb9bv"]
Sep 17 18:29:30.851769 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:29:30.851731    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" path="/var/lib/kubelet/pods/9a75af94-3ad6-4701-9ca0-f02b5b826301/volumes"
Sep 17 18:32:44.493236 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493556    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main"
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493569    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main"
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493592    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy"
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493599    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy"
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493656    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a75af94-3ad6-4701-9ca0-f02b5b826301" containerName="istio-proxy"
Sep 17 18:32:44.493838 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.493664    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aaab282-5173-41bd-86ab-fb77c4d1556c" containerName="main"
Sep 17 18:32:44.496315 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.496297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.498553 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.498528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-pd27s\""
Sep 17 18:32:44.498553 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.498545    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:32:44.498746 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.498578    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:32:44.498834 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.498819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 18:32:44.509632 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.509605    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:32:44.644478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.644478 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.644701 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmxq\" (UniqueName: \"kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.644701 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.644701 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.644701 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.644627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745252 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745252 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745252 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmxq\" (UniqueName: \"kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745642 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745811 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745811 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.745811 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.745767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.747594 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.747567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.748025 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.748007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:44.755450 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.755430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:32:44.765564 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.765534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:32:44.776618 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:44.776594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmxq\" (UniqueName: \"kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.683856 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.683805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.690213 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.690187    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.784463 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.784417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.784645 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.784529    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.789504 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.789482    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") DevicePath \"csi-f56969e8402876d0e9670a0806c74e119dd1a824f11702996a6c819e246732dc\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.887933 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.887902    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2b7e88d0f08fdc756cce5bfc50f5b573cede2d505463b07e3d2b15552285577b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:48.907690 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:48.907657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:49.009705 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:49.009616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-pd27s\""
Sep 17 18:32:49.018306 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:49.018261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:49.308986 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:49.308892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:32:49.313197 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:32:49.313161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fef9270_0047_4aa7_8b03_bff945c489fb.slice/crio-2d81454edbb51a351415abf008b1cdda10964d4d57a51ed3d4c6dac6ff599cb8 WatchSource:0}: Error finding container 2d81454edbb51a351415abf008b1cdda10964d4d57a51ed3d4c6dac6ff599cb8: Status 404 returned error can't find the container with id 2d81454edbb51a351415abf008b1cdda10964d4d57a51ed3d4c6dac6ff599cb8
Sep 17 18:32:50.099035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:50.098930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" event={"ID":"0fef9270-0047-4aa7-8b03-bff945c489fb","Type":"ContainerStarted","Data":"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"}
Sep 17 18:32:50.099035 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:50.098992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" event={"ID":"0fef9270-0047-4aa7-8b03-bff945c489fb","Type":"ContainerStarted","Data":"2d81454edbb51a351415abf008b1cdda10964d4d57a51ed3d4c6dac6ff599cb8"}
Sep 17 18:32:50.131479 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:50.131419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podStartSLOduration=6.131385668 podStartE2EDuration="6.131385668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:50.129450723 +0000 UTC m=+3545.787925719" watchObservedRunningTime="2026-09-17 18:32:50.131385668 +0000 UTC m=+3545.789860667"
Sep 17 18:32:51.019018 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:51.018973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:59.019229 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:59.019181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:32:59.020931 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:32:59.020896    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:09.019306 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:09.019249    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:19.019355 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:19.019301    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:29.019821 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:29.019770    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:39.019743 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:39.019644    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:45.707913 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:45.707794    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:33:45.716989 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:45.712050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:33:49.019141 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:49.019085    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:33:59.019700 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:33:59.019647    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:34:09.018839 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:34:09.018790    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:34:19.028933 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:34:19.028887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:34:19.037456 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:34:19.037427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:34:49.124605 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:34:49.124565    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw_0fef9270-0047-4aa7-8b03-bff945c489fb/main/0.log"
Sep 17 18:34:50.297677 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:34:50.297641    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:35:05.321362 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:05.321310    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" containerID="cri-o://c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53" gracePeriod=45
Sep 17 18:35:06.082758 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.082729    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:35:06.146587 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsmxq\" (UniqueName: \"kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.146814 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.146814 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.146814 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146764    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.146814 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.147038 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.147038 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.146883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache\") pod \"0fef9270-0047-4aa7-8b03-bff945c489fb\" (UID: \"0fef9270-0047-4aa7-8b03-bff945c489fb\") "
Sep 17 18:35:06.147232 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.147200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home" (OuterVolumeSpecName: "home") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:06.147372 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.147326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:06.149045 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.149015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm" (OuterVolumeSpecName: "dshm") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:06.149235 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.149214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq" (OuterVolumeSpecName: "kube-api-access-xsmxq") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "kube-api-access-xsmxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:06.149360 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.149333    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:06.156442 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.156413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada" (OuterVolumeSpecName: "kserve-pvc-source") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:35:06.159313 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.159277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fef9270-0047-4aa7-8b03-bff945c489fb" (UID: "0fef9270-0047-4aa7-8b03-bff945c489fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:06.248505 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248461    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248519    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") on node \"ip-10-0-128-61.ec2.internal\" "
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248533    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248543    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fef9270-0047-4aa7-8b03-bff945c489fb-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248551    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmxq\" (UniqueName: \"kubernetes.io/projected/0fef9270-0047-4aa7-8b03-bff945c489fb-kube-api-access-xsmxq\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.248702 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.248566    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fef9270-0047-4aa7-8b03-bff945c489fb-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:35:06.276566 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.276542    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada") on node "ip-10-0-128-61.ec2.internal"
Sep 17 18:35:06.349062 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.349019    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-51f82849-af5a-49c7-9a2a-c5f1395ecf1a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b3b08c642a1e1ada\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"csi-f56969e8402876d0e9670a0806c74e119dd1a824f11702996a6c819e246732dc\""
Sep 17 18:35:06.782386 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.782349    2577 generic.go:356] "Generic (PLEG): container finished" podID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerID="c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53" exitCode=0
Sep 17 18:35:06.782615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.782427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"
Sep 17 18:35:06.782615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.782425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" event={"ID":"0fef9270-0047-4aa7-8b03-bff945c489fb","Type":"ContainerDied","Data":"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"}
Sep 17 18:35:06.782615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.782555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" event={"ID":"0fef9270-0047-4aa7-8b03-bff945c489fb","Type":"ContainerDied","Data":"2d81454edbb51a351415abf008b1cdda10964d4d57a51ed3d4c6dac6ff599cb8"}
Sep 17 18:35:06.782615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.782579    2577 scope.go:117] "RemoveContainer" containerID="c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"
Sep 17 18:35:06.792055 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.792026    2577 scope.go:117] "RemoveContainer" containerID="c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"
Sep 17 18:35:06.792378 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:35:06.792362    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53\": container with ID starting with c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53 not found: ID does not exist" containerID="c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"
Sep 17 18:35:06.792491 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.792385    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53"} err="failed to get container status \"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53\": rpc error: code = NotFound desc = could not find container \"c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53\": container with ID starting with c76a1fa5229b75f6978911baeb65cdede5eca6a106236bc71d73b1fee5d1ce53 not found: ID does not exist"
Sep 17 18:35:06.810575 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.810535    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:35:06.817619 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.817581    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw"]
Sep 17 18:35:06.851456 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:06.851413    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" path="/var/lib/kubelet/pods/0fef9270-0047-4aa7-8b03-bff945c489fb/volumes"
Sep 17 18:35:07.030279 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:07.030230    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-5vnxw" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Sep 17 18:35:36.055076 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.055037    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:35:36.055688 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.055373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main"
Sep 17 18:35:36.055688 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.055384    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main"
Sep 17 18:35:36.055688 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.055475    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fef9270-0047-4aa7-8b03-bff945c489fb" containerName="main"
Sep 17 18:35:36.058108 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.058080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.061437 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.061180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-5z4rc\""
Sep 17 18:35:36.061437 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.061257    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:35:36.061437 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.061195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-rx7tf\""
Sep 17 18:35:36.061437 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.061195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:35:36.061437 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.061195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:35:36.067627 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.067595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:35:36.200939 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.200898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntbd\" (UniqueName: \"kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.201129 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.200963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.302317 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.302277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.302571 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.302373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntbd\" (UniqueName: \"kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.305060 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.305030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.311076 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.310981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntbd\" (UniqueName: \"kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.370184 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.370140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.651710 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.651682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:35:36.654561 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:35:36.654523    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3059ca5f_6a2e_4a8f_baf2_3ed4ac460725.slice/crio-8dc0712e5e080e5b7ad8520f447a4c061c0d3b326e19dcf0bdf0b77996f51248 WatchSource:0}: Error finding container 8dc0712e5e080e5b7ad8520f447a4c061c0d3b326e19dcf0bdf0b77996f51248: Status 404 returned error can't find the container with id 8dc0712e5e080e5b7ad8520f447a4c061c0d3b326e19dcf0bdf0b77996f51248
Sep 17 18:35:36.924258 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.924148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" event={"ID":"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725","Type":"ContainerStarted","Data":"a515bf8d5ca219850df159ac276a27e86a4240d62cce9c9ddbb978ba1cd1dc54"}
Sep 17 18:35:36.924258 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.924217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" event={"ID":"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725","Type":"ContainerStarted","Data":"8dc0712e5e080e5b7ad8520f447a4c061c0d3b326e19dcf0bdf0b77996f51248"}
Sep 17 18:35:36.924483 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.924334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:35:36.940169 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:35:36.940105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podStartSLOduration=0.940086426 podStartE2EDuration="940.086426ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:36.939818363 +0000 UTC m=+3712.598293363" watchObservedRunningTime="2026-09-17 18:35:36.940086426 +0000 UTC m=+3712.598561415"
Sep 17 18:36:07.930048 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:36:07.930012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:37:30.426552 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:30.426476    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:37:37.930053 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:37.930012    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:45.450147 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:45.450099    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main" containerID="cri-o://a515bf8d5ca219850df159ac276a27e86a4240d62cce9c9ddbb978ba1cd1dc54" gracePeriod=45
Sep 17 18:37:47.930700 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:47.930652    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:57.930376 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:57.930333    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:37:57.930843 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:37:57.930482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:38:07.929975 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:07.929883    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:15.657976 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.657940    2577 generic.go:356] "Generic (PLEG): container finished" podID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerID="a515bf8d5ca219850df159ac276a27e86a4240d62cce9c9ddbb978ba1cd1dc54" exitCode=0
Sep 17 18:38:15.658465 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.658023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" event={"ID":"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725","Type":"ContainerDied","Data":"a515bf8d5ca219850df159ac276a27e86a4240d62cce9c9ddbb978ba1cd1dc54"}
Sep 17 18:38:15.712484 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.712385    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:38:15.806375 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.806337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs\") pod \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") "
Sep 17 18:38:15.806604 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.806430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lntbd\" (UniqueName: \"kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd\") pod \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\" (UID: \"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725\") "
Sep 17 18:38:15.808754 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.808713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" (UID: "3059ca5f-6a2e-4a8f-baf2-3ed4ac460725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:15.808889 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.808748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd" (OuterVolumeSpecName: "kube-api-access-lntbd") pod "3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" (UID: "3059ca5f-6a2e-4a8f-baf2-3ed4ac460725"). InnerVolumeSpecName "kube-api-access-lntbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:15.906952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.906908    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.906952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:15.906949    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lntbd\" (UniqueName: \"kubernetes.io/projected/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725-kube-api-access-lntbd\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:38:16.663703 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.663665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w" event={"ID":"3059ca5f-6a2e-4a8f-baf2-3ed4ac460725","Type":"ContainerDied","Data":"8dc0712e5e080e5b7ad8520f447a4c061c0d3b326e19dcf0bdf0b77996f51248"}
Sep 17 18:38:16.663703 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.663696    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"
Sep 17 18:38:16.664275 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.663731    2577 scope.go:117] "RemoveContainer" containerID="a515bf8d5ca219850df159ac276a27e86a4240d62cce9c9ddbb978ba1cd1dc54"
Sep 17 18:38:16.685725 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.685681    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:38:16.689675 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.689643    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-xc29w"]
Sep 17 18:38:16.850972 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:16.850930    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" path="/var/lib/kubelet/pods/3059ca5f-6a2e-4a8f-baf2-3ed4ac460725/volumes"
Sep 17 18:38:32.669652 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.669610    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:38:32.670103 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.669928    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main"
Sep 17 18:38:32.670103 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.669940    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main"
Sep 17 18:38:32.670103 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.669998    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3059ca5f-6a2e-4a8f-baf2-3ed4ac460725" containerName="main"
Sep 17 18:38:32.672845 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.672815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.675600 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.675569    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:38:32.675770 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.675600    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:38:32.675770 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.675683    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:38:32.675770 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.675756    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-tdcxk\""
Sep 17 18:38:32.675935 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.675790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fbrr8\""
Sep 17 18:38:32.681419 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.681376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:38:32.737507 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.737475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvvd\" (UniqueName: \"kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.737690 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.737527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.838815 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.838776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.839022 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.838843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvvd\" (UniqueName: \"kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.841498 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.841464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.847998 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.847963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvvd\" (UniqueName: \"kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:32.985051 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:32.984951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:33.273486 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:33.273272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:38:33.276272 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:38:33.276227    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6a2bdb_b2bf_4847_ba69_3918838862d3.slice/crio-6274505a6920d41f18b5efdab2ad02ecaa46d789ddcdbda226e5cbbfe9d274c6 WatchSource:0}: Error finding container 6274505a6920d41f18b5efdab2ad02ecaa46d789ddcdbda226e5cbbfe9d274c6: Status 404 returned error can't find the container with id 6274505a6920d41f18b5efdab2ad02ecaa46d789ddcdbda226e5cbbfe9d274c6
Sep 17 18:38:33.755446 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:33.755383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" event={"ID":"3e6a2bdb-b2bf-4847-ba69-3918838862d3","Type":"ContainerStarted","Data":"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"}
Sep 17 18:38:33.755926 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:33.755511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:38:33.755926 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:33.755536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" event={"ID":"3e6a2bdb-b2bf-4847-ba69-3918838862d3","Type":"ContainerStarted","Data":"6274505a6920d41f18b5efdab2ad02ecaa46d789ddcdbda226e5cbbfe9d274c6"}
Sep 17 18:38:33.775817 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:33.775748    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podStartSLOduration=1.77573249 podStartE2EDuration="1.77573249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:33.773739957 +0000 UTC m=+3889.432214959" watchObservedRunningTime="2026-09-17 18:38:33.77573249 +0000 UTC m=+3889.434207487"
Sep 17 18:38:45.733621 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:45.733493    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:38:45.752533 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:38:45.740690    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:39:04.762369 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:39:04.762329    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:40:24.443152 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.443115    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:24.446068 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.446045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.448744 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.448718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:40:24.448894 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.448733    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:40:24.448984 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.448966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:40:24.449096 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.449073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-jlcr5\""
Sep 17 18:40:24.457810 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.457788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:24.531304 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rkn\" (UniqueName: \"kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.531499 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.531499 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.531499 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.531499 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.531646 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.531539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632163 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632336 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632336 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632336 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632520 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rkn\" (UniqueName: \"kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632520 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632697 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632760 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.632807 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.632761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.634737 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.634708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.634950 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.634930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.640698 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.640669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rkn\" (UniqueName: \"kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wcf4g\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:24.756858 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:24.756750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:25.032479 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:25.032449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:25.035222 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:40:25.035186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d1f861_63f4_4661_bf52_657535f254ea.slice/crio-583ae896ae1e9dcf69fe194312a21cd7dbac1e7427cf7e3ccf7a7506fb63189b WatchSource:0}: Error finding container 583ae896ae1e9dcf69fe194312a21cd7dbac1e7427cf7e3ccf7a7506fb63189b: Status 404 returned error can't find the container with id 583ae896ae1e9dcf69fe194312a21cd7dbac1e7427cf7e3ccf7a7506fb63189b
Sep 17 18:40:25.273779 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:25.273737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" event={"ID":"47d1f861-63f4-4661-bf52-657535f254ea","Type":"ContainerStarted","Data":"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"}
Sep 17 18:40:25.273973 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:25.273792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" event={"ID":"47d1f861-63f4-4661-bf52-657535f254ea","Type":"ContainerStarted","Data":"583ae896ae1e9dcf69fe194312a21cd7dbac1e7427cf7e3ccf7a7506fb63189b"}
Sep 17 18:40:25.293357 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:25.293231    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" podStartSLOduration=1.293214202 podStartE2EDuration="1.293214202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:25.29264118 +0000 UTC m=+4000.951116180" watchObservedRunningTime="2026-09-17 18:40:25.293214202 +0000 UTC m=+4000.951689202"
Sep 17 18:40:25.757210 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:25.757177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:34.757366 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:34.757313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:34.770001 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:34.769968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:35.330468 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:35.330435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:35.961791 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:35.961745    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:36.435787 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:36.435750    2577 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 18:40:36.435990 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:36.435829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs podName:47d1f861-63f4-4661-bf52-657535f254ea nodeName:}" failed. No retries permitted until 2026-09-17 18:40:36.935812842 +0000 UTC m=+4012.594287820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" (UID: "47d1f861-63f4-4661-bf52-657535f254ea") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 18:40:36.940564 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:36.940521    2577 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 18:40:36.940794 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:36.940620    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs podName:47d1f861-63f4-4661-bf52-657535f254ea nodeName:}" failed. No retries permitted until 2026-09-17 18:40:37.940605107 +0000 UTC m=+4013.599080084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" (UID: "47d1f861-63f4-4661-bf52-657535f254ea") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:37.338955    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:  > containerID="e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29" cmd=["/bin/sleep","15"]
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.339021    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" message="" err=<
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:  >
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:37.339035    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 17 18:40:37.339059 ip-10-0-128-61 kubenswrapper[2577]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" podUID="47d1f861-63f4-4661-bf52-657535f254ea" containerName="main" containerID="cri-o://e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"
Sep 17 18:40:37.339878 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.339072    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" podUID="47d1f861-63f4-4661-bf52-657535f254ea" containerName="main" containerID="cri-o://e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29" gracePeriod=60
Sep 17 18:40:37.694193 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.694154    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:37.850444 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4rkn\" (UniqueName: \"kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850666 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850666 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850490    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850666 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850666 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850666 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850613    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir\") pod \"47d1f861-63f4-4661-bf52-657535f254ea\" (UID: \"47d1f861-63f4-4661-bf52-657535f254ea\") "
Sep 17 18:40:37.850952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home" (OuterVolumeSpecName: "home") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:37.850952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850864    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-home\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:37.850952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:37.850952 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.850934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:37.852893 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.852868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn" (OuterVolumeSpecName: "kube-api-access-b4rkn") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "kube-api-access-b4rkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:40:37.852991 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.852866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:40:37.852991 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.852926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm" (OuterVolumeSpecName: "dshm") pod "47d1f861-63f4-4661-bf52-657535f254ea" (UID: "47d1f861-63f4-4661-bf52-657535f254ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:40:37.952256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.952152    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-tmp-dir\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:37.952256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.952194    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rkn\" (UniqueName: \"kubernetes.io/projected/47d1f861-63f4-4661-bf52-657535f254ea-kube-api-access-b4rkn\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:37.952256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.952203    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-dshm\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:37.952256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.952215    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d1f861-63f4-4661-bf52-657535f254ea-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:37.952256 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:37.952224    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d1f861-63f4-4661-bf52-657535f254ea-model-cache\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:40:38.335622 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.335528    2577 generic.go:356] "Generic (PLEG): container finished" podID="47d1f861-63f4-4661-bf52-657535f254ea" containerID="e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29" exitCode=0
Sep 17 18:40:38.335622 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.335573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"
Sep 17 18:40:38.335622 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.335600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" event={"ID":"47d1f861-63f4-4661-bf52-657535f254ea","Type":"ContainerDied","Data":"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"}
Sep 17 18:40:38.335883 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.335648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g" event={"ID":"47d1f861-63f4-4661-bf52-657535f254ea","Type":"ContainerDied","Data":"583ae896ae1e9dcf69fe194312a21cd7dbac1e7427cf7e3ccf7a7506fb63189b"}
Sep 17 18:40:38.335883 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.335673    2577 scope.go:117] "RemoveContainer" containerID="e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"
Sep 17 18:40:38.345335 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.345173    2577 scope.go:117] "RemoveContainer" containerID="e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"
Sep 17 18:40:38.345688 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:40:38.345560    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29\": container with ID starting with e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29 not found: ID does not exist" containerID="e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"
Sep 17 18:40:38.345688 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.345592    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29"} err="failed to get container status \"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29\": rpc error: code = NotFound desc = could not find container \"e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29\": container with ID starting with e3d545aad7f46fdc3dba7fa7fd3a7498731c5dc410e5838e9140b65d3f246f29 not found: ID does not exist"
Sep 17 18:40:38.363624 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.363582    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:38.370257 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.370217    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wcf4g"]
Sep 17 18:40:38.850810 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:40:38.850770    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d1f861-63f4-4661-bf52-657535f254ea" path="/var/lib/kubelet/pods/47d1f861-63f4-4661-bf52-657535f254ea/volumes"
Sep 17 18:41:13.626040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:13.625938    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:41:14.762749 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:14.762702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:24.762793 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:24.762750    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:28.647489 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:28.647443    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" containerID="cri-o://1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769" gracePeriod=45
Sep 17 18:41:34.762909 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:34.762858    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:34.763454 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:34.762972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:41:44.768587 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:44.768543    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:54.762542 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:54.762495    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:58.905891 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:58.905807    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:41:59.002610 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.002565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wvvd\" (UniqueName: \"kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd\") pod \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") "
Sep 17 18:41:59.002822 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.002636    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs\") pod \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\" (UID: \"3e6a2bdb-b2bf-4847-ba69-3918838862d3\") "
Sep 17 18:41:59.004937 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.004895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd" (OuterVolumeSpecName: "kube-api-access-7wvvd") pod "3e6a2bdb-b2bf-4847-ba69-3918838862d3" (UID: "3e6a2bdb-b2bf-4847-ba69-3918838862d3"). InnerVolumeSpecName "kube-api-access-7wvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:59.005077 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.004977    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e6a2bdb-b2bf-4847-ba69-3918838862d3" (UID: "3e6a2bdb-b2bf-4847-ba69-3918838862d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:59.104296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.104244    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6a2bdb-b2bf-4847-ba69-3918838862d3-tls-certs\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:41:59.104296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.104285    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wvvd\" (UniqueName: \"kubernetes.io/projected/3e6a2bdb-b2bf-4847-ba69-3918838862d3-kube-api-access-7wvvd\") on node \"ip-10-0-128-61.ec2.internal\" DevicePath \"\""
Sep 17 18:41:59.711550 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.711505    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerID="1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769" exitCode=0
Sep 17 18:41:59.711550 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.711538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" event={"ID":"3e6a2bdb-b2bf-4847-ba69-3918838862d3","Type":"ContainerDied","Data":"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"}
Sep 17 18:41:59.711774 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.711602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f" event={"ID":"3e6a2bdb-b2bf-4847-ba69-3918838862d3","Type":"ContainerDied","Data":"6274505a6920d41f18b5efdab2ad02ecaa46d789ddcdbda226e5cbbfe9d274c6"}
Sep 17 18:41:59.711774 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.711554    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"
Sep 17 18:41:59.711774 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.711630    2577 scope.go:117] "RemoveContainer" containerID="1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"
Sep 17 18:41:59.726532 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.726501    2577 scope.go:117] "RemoveContainer" containerID="1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"
Sep 17 18:41:59.726905 ip-10-0-128-61 kubenswrapper[2577]: E0917 18:41:59.726881    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769\": container with ID starting with 1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769 not found: ID does not exist" containerID="1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"
Sep 17 18:41:59.726962 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.726917    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769"} err="failed to get container status \"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769\": rpc error: code = NotFound desc = could not find container \"1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769\": container with ID starting with 1fc4a9545ad85dee109a7732c59586381dabbc98aafca7232b99abaf480d1769 not found: ID does not exist"
Sep 17 18:41:59.737775 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.737733    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:41:59.739830 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:41:59.739794    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-2j47f"]
Sep 17 18:42:00.851543 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:00.851501    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" path="/var/lib/kubelet/pods/3e6a2bdb-b2bf-4847-ba69-3918838862d3/volumes"
Sep 17 18:42:26.589615 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:26.589581    2577 ???:1] "http: TLS handshake error from 10.0.135.84:41010: EOF"
Sep 17 18:42:26.672499 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:26.672468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-d82ns_be6ec2e3-4301-4131-b718-69fb39c67bb7/discovery/0.log"
Sep 17 18:42:27.448296 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:27.448250    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-d82ns_be6ec2e3-4301-4131-b718-69fb39c67bb7/discovery/0.log"
Sep 17 18:42:28.328734 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:28.328697    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-qqzjm_32401980-5604-4d44-8188-1c5e904ce975/manager/0.log"
Sep 17 18:42:28.349734 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:28.349700    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-mdg6t_9cea7ecb-c655-4d21-b737-8c8a8e1985cb/manager/0.log"
Sep 17 18:42:28.442353 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:28.442319    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-cqkdm_7ddc4719-0aa1-4d32-980a-eeacf456ec0b/manager/0.log"
Sep 17 18:42:28.471142 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:28.471116    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wvl5v_09aebdaf-0437-4c63-9995-d7f7da1b5643/manager/0.log"
Sep 17 18:42:37.220408 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:37.220369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nplrw_acebde19-4743-4eaf-9037-c00785c8e0b1/global-pull-secret-syncer/0.log"
Sep 17 18:42:37.340935 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:37.340900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vgkv9_8296a988-6fdf-40e4-9b36-c2fa1b9d89e2/konnectivity-agent/0.log"
Sep 17 18:42:37.362124 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:37.362094    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-61.ec2.internal_d2516c3e83cd19c1935566a48ec58d2f/haproxy/0.log"
Sep 17 18:42:42.614850 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:42.614814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-qqzjm_32401980-5604-4d44-8188-1c5e904ce975/manager/0.log"
Sep 17 18:42:42.650533 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:42.650501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65dc8f76f5-mdg6t_9cea7ecb-c655-4d21-b737-8c8a8e1985cb/manager/0.log"
Sep 17 18:42:42.770293 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:42.770255    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-cqkdm_7ddc4719-0aa1-4d32-980a-eeacf456ec0b/manager/0.log"
Sep 17 18:42:42.836323 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:42.836281    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-wvl5v_09aebdaf-0437-4c63-9995-d7f7da1b5643/manager/0.log"
Sep 17 18:42:44.406908 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:44.406874    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2w9f5_171078fd-2465-47e0-80bd-6fa57d65baaa/node-exporter/0.log"
Sep 17 18:42:44.429205 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:44.429167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2w9f5_171078fd-2465-47e0-80bd-6fa57d65baaa/kube-rbac-proxy/0.log"
Sep 17 18:42:44.463861 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:44.463836    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2w9f5_171078fd-2465-47e0-80bd-6fa57d65baaa/init-textfile/0.log"
Sep 17 18:42:46.447069 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:46.447033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-h5lsq_dc988e16-a3cd-49c4-96bf-5ecc628daec7/networking-console-plugin/0.log"
Sep 17 18:42:46.902327 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:46.902295    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/2.log"
Sep 17 18:42:46.906943 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:46.906911    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cl69d_517ec90c-e584-4cd8-a7f0-f8a1e7207267/console-operator/3.log"
Sep 17 18:42:47.234085 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.233993    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"]
Sep 17 18:42:47.234325 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234313    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main"
Sep 17 18:42:47.234369 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234335    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main"
Sep 17 18:42:47.234369 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234348    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d1f861-63f4-4661-bf52-657535f254ea" containerName="main"
Sep 17 18:42:47.234369 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234354    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d1f861-63f4-4661-bf52-657535f254ea" containerName="main"
Sep 17 18:42:47.234511 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234417    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e6a2bdb-b2bf-4847-ba69-3918838862d3" containerName="main"
Sep 17 18:42:47.234511 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.234426    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="47d1f861-63f4-4661-bf52-657535f254ea" containerName="main"
Sep 17 18:42:47.238307 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.238284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.240636 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.240604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"kube-root-ca.crt\""
Sep 17 18:42:47.241232 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.241208    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mv6cb\"/\"default-dockercfg-nvlfk\""
Sep 17 18:42:47.241358 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.241208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mv6cb\"/\"openshift-service-ca.crt\""
Sep 17 18:42:47.254005 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.253969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"]
Sep 17 18:42:47.394784 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.394736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-proc\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.394784 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.394789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4sl\" (UniqueName: \"kubernetes.io/projected/9f204584-69da-4370-8730-46553efe2254-kube-api-access-dx4sl\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.395048 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.394850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-podres\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.395048 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.394896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-lib-modules\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.395048 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.394948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-sys\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496043 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.495947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-sys\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496043 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.495999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-proc\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496043 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4sl\" (UniqueName: \"kubernetes.io/projected/9f204584-69da-4370-8730-46553efe2254-kube-api-access-dx4sl\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-podres\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-lib-modules\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-sys\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-proc\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-lib-modules\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.496561 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.496216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f204584-69da-4370-8730-46553efe2254-podres\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.504493 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.504467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4sl\" (UniqueName: \"kubernetes.io/projected/9f204584-69da-4370-8730-46553efe2254-kube-api-access-dx4sl\") pod \"perf-node-gather-daemonset-w7w7q\" (UID: \"9f204584-69da-4370-8730-46553efe2254\") " pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.548276 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.548234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:47.821340 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.821243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"]
Sep 17 18:42:47.824645 ip-10-0-128-61 kubenswrapper[2577]: W0917 18:42:47.824610    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f204584_69da_4370_8730_46553efe2254.slice/crio-7aa8102401633c4771f6e66b3bf0bbe6d4cc6b3a22d16d12a3b44c38a06f80d5 WatchSource:0}: Error finding container 7aa8102401633c4771f6e66b3bf0bbe6d4cc6b3a22d16d12a3b44c38a06f80d5: Status 404 returned error can't find the container with id 7aa8102401633c4771f6e66b3bf0bbe6d4cc6b3a22d16d12a3b44c38a06f80d5
Sep 17 18:42:47.932202 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:47.932162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q" event={"ID":"9f204584-69da-4370-8730-46553efe2254","Type":"ContainerStarted","Data":"7aa8102401633c4771f6e66b3bf0bbe6d4cc6b3a22d16d12a3b44c38a06f80d5"}
Sep 17 18:42:48.938312 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:48.938270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q" event={"ID":"9f204584-69da-4370-8730-46553efe2254","Type":"ContainerStarted","Data":"c29736c31210e8e9ce17ec2c1ddd63fba4550a085913594e927f9211ffedf9d2"}
Sep 17 18:42:48.938857 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:48.938384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:48.955218 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:48.955165    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q" podStartSLOduration=1.9551474660000001 podStartE2EDuration="1.955147466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:48.954139742 +0000 UTC m=+4144.612614742" watchObservedRunningTime="2026-09-17 18:42:48.955147466 +0000 UTC m=+4144.613622469"
Sep 17 18:42:49.112147 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:49.112116    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mh8gs_e0526a7e-2c2b-4892-ae86-883f3253ae5a/dns/0.log"
Sep 17 18:42:49.134348 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:49.134298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mh8gs_e0526a7e-2c2b-4892-ae86-883f3253ae5a/kube-rbac-proxy/0.log"
Sep 17 18:42:49.238111 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:49.238023    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7xgx_263b9c4f-3fea-44fe-8f45-2da654828fca/dns-node-resolver/0.log"
Sep 17 18:42:49.811911 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:49.811869    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cfzd2_060d1280-c356-48bc-8a89-4ddb085bf775/node-ca/0.log"
Sep 17 18:42:50.834721 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:50.834677    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-d82ns_be6ec2e3-4301-4131-b718-69fb39c67bb7/discovery/0.log"
Sep 17 18:42:51.438749 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:51.438711    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpqnj_9f41ff3e-865a-44f8-88ae-efa0b62504f4/serve-healthcheck-canary/0.log"
Sep 17 18:42:52.091364 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:52.091333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qg5lf_e3415f11-9bc0-4a2b-8ea4-1275704ef3ba/kube-rbac-proxy/0.log"
Sep 17 18:42:52.117293 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:52.117253    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qg5lf_e3415f11-9bc0-4a2b-8ea4-1275704ef3ba/exporter/0.log"
Sep 17 18:42:52.140480 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:52.140449    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qg5lf_e3415f11-9bc0-4a2b-8ea4-1275704ef3ba/extractor/0.log"
Sep 17 18:42:54.953825 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:54.953793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mv6cb/perf-node-gather-daemonset-w7w7q"
Sep 17 18:42:55.048913 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:55.048875    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-2cwgm_77a6398b-dce4-4c15-8848-dfc10187952b/openshift-lws-operator/0.log"
Sep 17 18:42:56.031578 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:56.031547    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-gzlsj_cfae0daa-f955-4c8d-a10b-b12e93762c25/server/0.log"
Sep 17 18:42:56.391464 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:56.391361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-d82ns_be6ec2e3-4301-4131-b718-69fb39c67bb7/discovery/0.log"
Sep 17 18:42:56.643533 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:56.643436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-pjt6r_28c081b6-9bd3-48ae-bb97-27f08e316b50/manager/0.log"
Sep 17 18:42:57.594378 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:42:57.594341    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-d82ns_be6ec2e3-4301-4131-b718-69fb39c67bb7/discovery/0.log"
Sep 17 18:43:03.033040 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.033005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:43:03.058546 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.058510    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/egress-router-binary-copy/0.log"
Sep 17 18:43:03.081595 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.081563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/cni-plugins/0.log"
Sep 17 18:43:03.105128 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.105092    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/bond-cni-plugin/0.log"
Sep 17 18:43:03.128332 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.128298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/routeoverride-cni/0.log"
Sep 17 18:43:03.152187 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.152152    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/whereabouts-cni-bincopy/0.log"
Sep 17 18:43:03.173760 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.173724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lg9m4_9c18a0f8-b17b-4da8-82f0-e4282e527a70/whereabouts-cni/0.log"
Sep 17 18:43:03.380658 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.380620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dq6jm_2fbabe68-32c9-4772-8316-ccc2d0c949fa/kube-multus/0.log"
Sep 17 18:43:03.543804 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.543774    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gvfrz_9d866f17-0e45-46eb-96c2-3638d706960c/network-metrics-daemon/0.log"
Sep 17 18:43:03.565300 ip-10-0-128-61 kubenswrapper[2577]: I0917 18:43:03.565269    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gvfrz_9d866f17-0e45-46eb-96c2-3638d706960c/kube-rbac-proxy/0.log"